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DEHNITIONS  OF  KEY  AIR  FORCE  TERMS  AND  ACRONYMS 

Activity:  Tenn  used  in  the  Air  Force  to  describe  separate  business  operations.  For  example,  the 
bowling  center,  the  Officer’s  Club,  and  lodging  are  all  activities. 

AF :  Acronym  for  Air  Force 

AFB :  Acronym  for  Air  Force  Base 

Appropriated  Fund  (APF)  civilians:  Government  employees  whose  salaries  are  tax  dollar  funded. 

Civilian:  For  this  project  a  civilian  indicates  a  government  employee  that  is  not  a  military  member 
on  active  duty. 

Consulting  companies:  Term  used  in  this  project  to  describe  independent  mystery  shopping 
companies.  These  companies’  primary  business  is  providing  mystery  shopping  services. 

Family  member:  People  who  are  family  members  of  any  type  of  government  employee  including 
retired  military. 

Flight:  Air  Force  organizational  level  consisting  of  several  similar  activities.  For  example, 
lodging,  military  dining  facilities,  fitness  centers,  libraries,  and  mortuary  affairs  combine  to  make 
the  Combat  Support  Flight. 

Non-appropriated  Fund  (NAF)  civilians:  Government  employees  whose  salaries  are  funded  from 
activity  revenues.  For  example,  money  that  customers  pay  for  foodservice  and  lodging  is  used  to 
fund  NAF  salaries. 

Program  manager:  Generic  job  title  given  to  the  person  who  manages  the  Air  Force  mystery 
shopping  program. 

Program  designer:  Generic  job  title  of  the  person  who  designed  the  Air  Force  mystery  shopping 
program. 

Services:  Generic  term  used  to  refer  to  activities  and  functions  performed  by  a  Services  Division. 

Services  Division:  Air  Force  organizational  level  normally  consisting  of  five  flights.  The  Services 
Division  at  Wright-Patterson  AFB  consists  of  over  60  activities  with  more  than  1,000  employees. 
Divisions  with  military  bosses  are  called  squadrons. 

Shop:  For  this  project,  a  shop  refers  to  a  single  mystery  shop  completed  by  either  a  company 
employee  or  an  independent  contractor. 

Shopper:  The  person  who  performs  mystery  shops 
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CHAPTER  I 
INTRODUCTION 

Statement  of  Problem 

Base  level  Air  Force  Services’  lodging  and  foodservice  activities  use  limited  service 
quality  measurement  tools  to  determine  customer  perceptions  of  service  quality.  These  tools, 
specifically  management  observation  and  customer  comment  cards,  do  not  provide  a  complete 
picture  of  service  quality.  Other  service  quality  measurement  methods  such  as  mystery  shopping 
are  rarely  used.  Bases  do  not  consider  using  mystery  shopping  programs  because  of  the 
significant  resources  required  to  start  the  program.  This  project  designed  and  tested  a  mystery 
shopper  program  for  the  US  Air  Force  foodservice  and  lodging  programs  at  Wright-Patterson 
AFB,  OH  as  a  prototype  for  other  bases. 

Background 

The  use  of  mystery  shoppers  to  evaluate  customer  service  experiences  is  widely  used  in 
commercial  foodservice  and  hotel  industries.  Many  companies  “shop”  their  properties  with  in- 
house  programs  while  others  contract  an  independent  company  to  perform  the  shopping. 

US  Air  Force  Services  activities  such  as  foodservice  and  lodging  have  increasingly 
focused  on  improving  customer  service  over  the  past  several  years.  Different  tools  used  by  the 
Air  Force  to  measure  service  quality  include  management  observation  and  customer  comment 
cards.  Both  of  these  methods  have  limited  application  due  to  measurement  biases.  As  a  result, 
there  is  a  need  to  measure  service  quality  in  an  unbiased  manner  to  help  identify  what  is  working 
well  and  what  areas  needs  attention.  A  fully  integrated  mystery  shopper  program  combined  with 
other  methods  of  evaluation  will  allow  Services  squadrons  and  divisions  to  better  measure  their 
service  quality,  recognize  employees,  and  take  immediate  action  to  fix  problems  as  they  occur. 
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US  Air  Force  Services  did  not  have  a  standard  mystery  shopping  program  in  place  before 
this  program  was  developed.  A  few  bases  were  performing  mystery  shopping  activities  on  an 
irregular  and  informal  basis.  One  of  the  biggest  obstacles  to  using  mystery  shoppers  was  the 
initial  effort  required  to  set-up  a  program.  In  addition,  bases  could  not  take  advantage  of 
commercial  mystery  shopping  services  because  many  shoppers  were  not  authorized  to  use  base 
facilities. 

Project  Overview 

This  project  designed,  set  up,  and  tested  a  prototype  mystery  shopping  program  at 
Wright-Patterson  Air  Force  Base  in  Dayton,  Ohio.  The  program  included  all  evaluation  forms, 
training  materials,  marketing  materials,  recruiting  materials,  a  shopper  tracking  database,  a 
results  analysis  database,  a  recognition  program,  a  budget,  and  an  annual  schedule  of  shops  for 
each  activity. 

This  mystery  shopping  program  was  designed  to  be  flexible  so  that  other  bases  could 
tailor  the  program  to  focus  on  the  evaluation  criteria  most  important  to  their  operations.  This 
program  evaluated  customer  service,  product  quality,  cleanliness,  facility  maintenance  and 
timeliness  of  service.  In  addition,  the  mystery  shopper  program  validated  customer  service 
training,  squadron  service  standards,  and  recognized  employees  for  outstanding  customer 
service. 

Fifty  shops  were  scheduled  to  test  the  mystery  shopping  program.  Ten  shops  were 
scheduled  for  the  Wright-Patterson  Inns  lodging  operation,  a  650-room  lodging  operation,  and  a 
total  of  forty  shops  were  scheduled  in  twenty  different  foodservice  operations  located  throughout 
Wright-Patterson  AFB.  Twenty-one  common  areas  were  rated  similarly  in  all  facilities  to 
provide  overall  results  for  the  entire  Services  Division.  Each  shopper  completed  a  program 
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critique  evaluating  the  training,  shopping  guide,  forms,  and  overall  program.  The  results  from 
these  critiques  combined  with  inputs  from  the  mystery  shopping  program  manager  and  designer 
were  used  to  evaluate  the  program  and  make  changes  as  necessary. 

In  addition,  this  project  informally  interviewed  five  hotel  chains,  five  restaurant  chains, 
and  five  independent  mystery  shopping  companies  to  identify  industry  standards  for  mystery 
shopping  recruiting,  training,  areas  of  evaluation,  frequency,  recognition,  and  costs.  These 
results  were  compared  to  the  decisions  made  designing  the  Air  Force  program  for  each  of  these 
areas. 
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Objectives  of  Study 

There  are  several  components  to  that  make  up  this  research  project.  To  help  differentiate 
them  a  list  of  objectives  are  provided.  These  objectives  are  listed  in  the  order  they  will  be 
accomplished. 

Objective  1: 

To  investigate  industry  mystery  shopping  standards  to  include  frequency,  cost,  and 
recognition 

Objective  2: 

To  design  a  prototype  mystery  shopper  program  for  Air  Force  Services  that  includes 
recruiting  and  training  of  shoppers,  recognition  of  employees,  cost  budgeting  and 
tracking,  and  results  tracking  and  analysis. 

Objective  3: 

To  test,  evaluate,  and  modify  the  Air  Force  Services  mystery  shopper  program 
Objective  4: 

Ensure  the  mystery  shopper  program  is  flexible  and  can  be  modified  easily  by  each  base 
using  the  program 
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CHAPTER  II 
LITERATURE  REVIEW 

Defining  Service  Quality 

“One  of  the  basic  tenets  of  the  quality  movement  is  that  the  customer  decides  what 
quality  is”  (Massnick,  1997,  p.  34).  However,  since  service  quality  is  intangible  and  means 
different  things  to  different  customers,  it  is  often  defined  in  terms  of  customer  perceptions  and 
expectations  in  the  service  quality  literature.  In  their  1985  article,  Parasuraman,  Zeithamal,  and 
Berry  defined  service  quality  as  the  amount  of  difference  between  customer’s  expectations  of  the 
service  and  their  perceptions  of  the  service  performance.  This  definition  has  been  frequently 
cited  and  corresponds  with  many  other  authors’  definitions.  Edvardsson,  Thomasson  and 
Overveit  word  it  this  way  “customer-perceived  quality  is  often  defined  as  the  relationship 
between  the  customer’s  expectations  of  the  service  and  his  or  her  perception  of  the  service 
received”  (1994,  p.  1). 

Importance  of  Service  Quality 

A  study  conducted  by  The  Forum  Corporation  revealed  that  nearly  70  percent  of  the 
identifiable  reasons  customers  left  companies  had  more  to  do  with  the  quality  of  service  than  the 
quality  of  the  product  (Massnick,  1997).  In  a  separate  study  conducted  in  1989  by  Gallup,  the 
American  Society  for  Quality  Control  (ACQC)  found  that  American  executives  ranked 
improvement  to  service  quality  as  the  most  important  means  to  improve  customer  satisfaction 
(Edvardsson  et  al.,  1994).  Air  Force  Services  also  places  a  strong  emphasis  on  measuring  and 
improving  service  quality. 

Keeping  customers  is  very  important,  not  just  to  prevent  reduced  revenue  from  lost 
customers,  but  also  because  of  the  potential  negative  influence  it  has  on  other  customers. 
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The  U.S.  Department  of  Commerce  has  found  that  more  than  90  percent  of  dissatisfied 
customers  will  simply  drift  over  to  the  competition,  though  not  always  silently.  The 
study  further  revealed  that  the  dissatisfied  customer  will  voice  his  displeasure  to 
sometimes  as  many  as  nine  of  his  friends,  relatives,  and  other  acquaintances  who  rely  on 
word-of-mouth  advertising  in  choosing  where  they  spend  their  money  (Bode,  1993, 

p.66). 

There  is  also  the  increased  expense  of  gaining  new  customers,  which  costs  about  five  times  more 
than  keeping  current  customers.  In  addition,  it  costs  about  ten  times  more  to  get  the  lost 
customer  to  return  (Massnick,  1997). 

Service  quality  is  important  because  it  directly  affects  the  satisfaction  level  of  customers 
zuid  in-tum,  the  satisfied  and  dissatisfied  customers  have  a  direct  affect  on  profits.  Keeping 
customers  and  keeping  the  customers  satisfied  is  especially  important  to  hospitality  companies 
because  such  a  large  part  of  their  “product”  is  service.  The  question  that  arises  is  how  do 
hospitality  companies  maintain  satisfied  customers?  “Sustained  and  continuous  quality 
improvement  is  not  possible  without  measures  of  quality.  Quality  measurement  is  probably  the 
most  important  technique  for  a  service  aiming  for  more  than  a  superficial  improvement” 
(Edvardsson  et  al.,  1994,  p.  178).  Customers’  high  expectations  and  the  increased  level  of 
competition  in  the  hospitality  industry  make  service  quality  measurement  more  important  than 
ever  for  companies  to  stay  competitive  (Ford  and  Bach,  1997).  Restaurants  and  hotels  are  no 
exception  and  must  find  ways  to  discover  their  customers’  opinions  of  their  service  quality. 
Measuring  Service  Quality 

There  are  many  aspects  that  customers  consider  when  perceiving  the  quality  of  a  service 
such  as  quick  service,  quality  product,  and  fun  atmosphere.  Since  each  customer  values  these 
aspects  differently,  it  is  difficult  to  determine  which  aspect  is  the  most  important.  In  their  service 
quality  research,  Parasuraman  et  al.  (1988)  identified  five  domains  that  categorize  the  different 
criteria  that  consumers  use  to  assess  service  quality.  These  domains  are  tangibles,  reliability. 
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responsiveness,  assurance,  and  empathy.  These  domains  can  be  used  to  ensure  that  the 
measurements  of  customer’s  service  quality  perceptions  are  equal  for  the  different  areas  that 
customers  deem  important. 

Questions  designed  to  measure  service  quality  can  be  designed  as  either  quantitative  or 
qualitative.  The  quantitative  measurements  obtain  objective  facts  such  as  the  number  of  rings 
before  a  phone  is  answered  and  whether  or  not  employees  are  wearing  a  nametag.  The 
qualitative  measurements  obtain  less  precise  verbal  information  from  the  subjects  which  helps 
managers  better  understand  customer’s  expectations  and  requirements  (Edvardsson  et  al.,  1994). 
Quantitative  and  qualitative  methods  complement  each  other  and  a  combination  of  both  should 
be  used  to  measure  service  quality  (Edvardsson  et  al,  1994,  p.l81).  Executives  from  ten  major 
mystery  shopping  companies  agreed  that  there  should  be  a  mix  of  objective  and  subjective 
questions  to  measure  service  quality.  They  also  felt  that  the  majority  of  questions  should 
measure  objective  aspects  (“Mystery  shoppers  resource,”  1994). 

Methods  Used  to  Measure  Service  Quality 

There  are  a  number  of  methods  that  are  used  to  measure  this  critical  element  of  the 
hospitality  industry.  These  methods  include  management  observation,  employee  feedback 
programs,  comment  cards,  mail  surveys,  on-site  personal  interviews,  telephone  interviews,  focus 
groups  and  mystery  shoppers.  “Selecting  the  best  method  requires  balancing  the  organization’s 
strategic  goals  and  the  cost  of  achieving  them”  (Ford  and  Bach,  1997,  p.  83).  The  two  most 
commonly  used  techniques  used  by  base  level  Air  Force  Services  are  management  observation 
and  customer  comment  cards. 

Many  of  the  available  methods  have  positive  and  negative  attributes.  For  example, 
management  observation  allows  the  manager  to  see  first  hand  what  is  going  on  without  having  to 
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interpret  the  perceptions  of  an  employee  or  customer.  On  the  other  hand,  many  employees  are 
on  their  “good  behavior”  when  the  manager  is  watching,  which  reduces  the  effectiveness  of  this 
method.  Use  of  customer  comment  cards  is  another  popular  technique.  The  benefits  include 
moderate  cost  of  data  collection  and  an  indication  that  the  company  is  interested  in  the 
customer’s  opinions  on  service  quality  (Ford  and  Bach,  1997).  The  drawbacks  of  conunent  cards 
include  a  self-selected  sample  of  customers,  which  is  not  statistically  representative,  and  that 
comment  cards  generally  reflect  the  opinions  of  customers  that  are  either  extremely  dissatisfied 
or  extremely  satisfied  with  their  service.  In  addition,  the  customer  may  only  report  the  worst 
items  that  they  did  not  like,  leaving  other  important  information  uncollected.  How  does  a 
company  get  a  customer’s  unbiased  assessment  without  having  employees  react  differently  than 
normal  (Ford  and  Bach,  1997;  Brown,  1989)? 

Mystery  Shopping 

One  method  is  mystery  shopping.  “Mystery  shopping  provides  an  impartial  guest’s  eye 
view  of  the  hospitality  operation”  (Stefanelli,  1994,  p.  17).  Mystery  shoppers  are  people  hired 
and  trained  (sometimes  after  the  first  shop)  to  use  a  company’s  services  and  record  their 
observations  (both  good  and  bad)  about  what  they  see.  They  provide  an  objective  perspective 
and  look  at  the  restaurant  or  hotel  with  fresh  eyes.  The  idea  is  to  “catch  people  doing  things 
right”  (Kelly,  1997,  p.  16).  The  employees  do  not  know  who  these  shoppers  are  and  therefore 
treat  them  the  same  as  any  other  customer.  However,  employees  do  know  that  every  customer 
could  be  a  potential  mystery  shopper,  and  the  areas  that  mystery  shoppers  are  evaluating.  This 
knowledge  motivates  the  employee  to  focus  on  the  most  important  aspects  of  the  business. 
“People  (employees)  do  what  you  inspect,  not  what  you  expect”  (Brown,  1989,  p.  136).  Don 
Bode  of  the  Consumer  Research  Group  believes  that  it  is  the  “combination  of  continual. 
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unbiased,  anonymous  ‘Positive  Customer  feedback’  [sic]  that  gives  the  assessment  its  optimal 
value”  (1993,  p.  66). 

“The  typical  shopping  process  is  an  on-site  visit  that  includes  an  objective  analysis  of 
service,  facility,  and  product  quality”  (Stefanelli,  1994,  p.  17).  After  the  visit,  the  shopper  writes 
a  detailed  report  on  the  visit  for  management  to  use  to  improve  quality  of  service.  Limitations  of 
mystery  shopping  include  recall  bias,  motivation  of  the  shopper,  a  moderate  to  high  cost  and  a 
potentially  statistically  invalid  sample  due  to  the  limited  number  of  shops  (Ford  and  Bach,  1997). 
“If  used  correctly,  a  mystery  shopping  program  will  yield  many  dividends  for  hospitality 
companies.  The  cost  of  mystery  shoppers  seems  small  compared  to  the  benefits  derived” 
(Stefanelli,  p.  18). 

There  are  no  research  articles  addressing  the  effectiveness  of  mystery  shopping,  however 
an  unpublished  study  conducted  over  a  six  year  period  from  1992  through  1997  by  Dr.  Joe  La 
Lopa,  Purdue  University,  and  the  Wales  Tourism  Board  found  significant  improvement  in  hotel 
properties  that  were  mystery  shopped  during  this  period.  Although  there  are  other  possible 
factors  that  could  have  contributed  to  this  improvement,  such  as  increased  competition  and 
higher  customer  expectations,  the  results  still  positively  support  mystery  shopping.  Other  than 
this  study,  the  only  indication  of  the  effectiveness  of  mystery  shopping  is  the  widespread  use  of 
the  method  in  the  service  industry.  There  are  over  400  independent  companies  that  provide 
mystery  shopping  services  (S.  Snedegar,  personal  communication,  June  2, 1998).  In  addition, 
many  companies  operate  in-house  programs. 

Air  Force  Services  operates  hospitality  businesses  to  serve  the  needs  of  the  military 
community.  These  programs  have  the  same  requirement  to  assess  service  quality  as  their 
commercial  counterparts.  Because  Air  Force  programs  can  only  serve  authorized  patrons,  many 
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Air  Force  bases  have  been  reluctant  to  utilize  mystery  shopper  programs  because  it  is  difficult  to 
hire  a  contractor  or  create  a  program  from  the  ground  up.  As  a  result,  there  is  not  a  standard  Air 
Force  mystery  shopper  program. 

Mystery  Shopper  Industry  Standards 

Since  there  is  little  research  on  mystery  shopping,  especially  concerning  food  service  and 
lodging,  this  project  identified  industry  standards  to  better  understand  the  typical  cost,  frequency, 
and  recognition  of  mystery  shopper  programs.  These  standards  are  derived  from  foodservice, 
lodging,  and  independent  mystery  shopping  companies  based  on  their  experience  v^^ith  their 
mystery  shopping  programs.'  Many  of  these  companies  developed  their  own  mystery  shopper 
programs  by  trial  and  error.  They  had  to  discover  the  best  questions,  frequency,  payment  and 
recognition  program  to  use  for  their  program.  There  are  two  different  methods  companies  use  to 
conduct  mystery  shopping:  in-house  and  contract. 

In-house  Programs 

Companies  that  use  in-house  mystery  shopping  programs  usually  conduct  all  mystery 
shopping  duties  with  internal  personnel.  The  corporate  office  sets  up  the  shopping  criteria, 
schedules  the  shops,  and  tracks  results  without  the  aid  of  independent  contractors.  Shoppers  are 
generally  regional  managers  or  corporate  personnel  who  are  generally  unknown  at  the  property 
level  (Stefanelli,  1994).  For  example,  Arby’s,  Inc.  created  an  in-house  program  called  “Catch 
Me  Doing  Something  Right”  after  using  independent  services.  Joe  Langtau,  Senior  Vice 
President  of  Operations,  explains,  “One  of  the  concerns  is  the  quality  of  folks  that  do  the 
evaluating.  We  had  problems  with  consistency  and  accuracy  of  evaluations”  (Thompson,  1993, 
p.  26).  Domino’s  Pizza  was  another.  They  also  switched  from  using  a  contract-shopping 
program  to  an  in-house  program  because  they  felt  that  for  the  cost,  they  could  get  better  feedback 


11 


from  internal  personnel.  They  use  area  supervisors  and  managers  to  call  their  own  stores  from 
friends’  houses  and  then  evaluate  the  service  and  product  quality  while  remaining  anonymous 
(Thompson). 

While  some  companies  want  shoppers  with  extensive  knowledge  about  the  business, 
other  companies  such  as  Sholodge  Franchise  Systems,  the  University  of  Kentucky  and  the 
University  of  Southern  California,  recruit  regular  customers  which  are  usually  less  informed  to 
be  their  shoppers.  They  can  provide  a  different  view  from  the  in-house  employee’s  perspective 
and  give  an  added  boost  to  public  relations,  but  it  also  adds  more  work  to  train  and  track  these 
shoppers  (“At  9  campus  facilities,”  1997;  Snedegar,  1998;  “Guest  ‘shoppers’”,  1996;  Quinton, 
1991) 

In-house  programs  have  their  advantages  and  disadvantages.  The  advantages  include  a 
closer  control  over  when  and  how  the  shops  are  conducted.  In  addition  they  can  receive  the 
feedback  more  quickly,  especially  if  internal  employees  are  conducting  the  shops.  They  can 
make  sure  that  shoppers  are  properly  trained  and  focus  on  the  right  arezis.  Some  of  the 
disadvantages  include  the  extra  time  and  money  required  recruiting,  training  and  tracking 
shoppers.  The  labor  expense  of  an  internal  employee  to  shop  a  property  will  usually  cost  more 
than  hiring  an  independent  shopper.  Internal  employees  usually  receive  an  hourly  wage  and 
reimbursement  for  the  meal  or  room  charges,  while  an  independent  shopper’s  compensation  is 
normally  only  the  meal  or  room  reimbursement.  A  regional  manager  conducting  a  shop  on  one 
his  or  her  own  properties  may  have  a  bias,  either  positive  or  negative,  toward  that  property 
(Thompson,  1993;  Brown,  1989). 
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Contract  Programs 

The  other  way  to  have  a  mystery  shopping  program  is  hiring  an  independent  research 
consultant.  This  is  an  extremely  fast  growing  industry  because  of  the  low  overhead  and  start-up 
costs.  Many  companies  use  the  Internet  to  recruit  shoppers,  distribute  questionnaires,  and 
receive  completed  shopping  reports.  There  are  at  least  400  companies  that  provide  shopping 
services  and  thousands  of  independent  shoppers  that  will  work  through  one  of  these  companies 
or  directly  for  the  company  wanting  the  shops.  Elrick  and  Lavidge  Mystery  Shopping,  for 
example,  maintains  a  database  of  over  25,000  shoppers  across  the  country  (Kephart,  1996). 

Some  other  companies  that  provide  mystery  shopping  services  include  T.I.P.S.,  Perks,  A  Closer 
Look,  and  Guest  Perception,  Inc.  to  name  a  few  (Thompson,  1993;  Cook,  1998;  “Mystery 
shoppers,”  1998). 

These  companies  provide  the  entire  shopping  service  including  questionnaire  design, 
recruiting,  training,  scheduling,  and  payment  of  the  shoppers.  Companies  who  hire  these 
services  still  need  to  be  involved  to  ensure  the  right  areas  are  shopped,  the  information  is 
properly  utilized,  and  recognition  of  employees  is  provided  (Thompson,  1993). 

Au  Bon  Pain  originally  created  their  own  in-house  program,  but  switched  to  an 
independent  company  because  it  was  too  difficult  to  train  shoppers  (Thompson,  1993).  [Note:  In 
a  telephone  interview  conducted  withAu  Bon  Pain  in  June  of 1998,  they  indicated  they  had 
stopped  using  mystery  shopping  all  together].  It  is  a  time  consuming  process  to  continuously 
recruit  and  train  shoppers  to  ensure  a  firesh  set  of  eyes  are  evaluating  the  company’s  products  and 
services.  The  advantages  of  using  a  contract  company  include  saving  company  time  and  money. 
Most  hospitality  companies  using  mystery  shopping,  prefer  to  hire  specialized  outside 
contractors  (Stefanelli,  1994).  The  disadvantages  include  losing  direct  control  over  when  and 
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how  the  shops  are  conducted.  Although  a  contract  may  specify  the  number  of  shops  that  will  be 
performed  during  a  certain  period  of  time,  the  actual  day  the  shop  is  completed  depends  of  the 
availability  of  a  shopper.  The  company  loses  control  over  how  the  shopper  is  trained  to  perform 
the  visit,  which  effects  the  quality  of  feedback.  Some  companies  believe  that  only  permanent 
employees  working  in  the  hospitality  industry  understand  the  business  and  can  be  trusted  with 
shopping  duties  (Stefanelli,  1994).  In  addition,  the  feedback  is  generally  slower  getting  back  to 
the  company. 

In  general,  larger  companies  should  consider  an  outside  contract.  “Mystery  shoppers  are 
best  or  most  often  used  by  companies  whose  outlets  are  spread  all  over  the  country  and  need  to 
ensure  that  quality  and  service  is  [sic]  consistent  at  all  outlets”  says  Bemyce  Hayes,  President  of 
Bemett  Research  Services  (Kephart,  1996,  p.  1-2).  Brian  Quinton  (1991)  agrees  advising  that  a 
small  concentrated  chain  can  use  an  in-house  program,  but  when  the  company  is  spread  over  two 
or  three  cities  or  states  then  an  outside  company  may  be  the  best  solution. 

Areas  of  Evaluation 

Mystery  shoppers  can  only  evaluate  the  restaurant  or  hotel  from  a  guest’s  perspective. 

As  a  result,  they  can  not  evaluate  the  profit/loss  ratio,  FDA  standards,  and  other  behind-the-scene 
standards  (Thompson,  1993).  But  mystery  shopping  programs  can  assess  employee 
performance,  service  quality,  product  quality,  cleanliness,  decor,  ambiance,  security,  suggestive 
selling,  training  programs  and  overall  experience.  (Cook,  1998;  Kelly,  1997;  Stafanelli,  1994; 
Thompson) 

“The  composition  of  the  questionnaire  is  critical  to  the  success  of  a  mystery  shopping 
program”  (Thompson,  1993,  p.  27).  If  an  outside  consultant  is  used,  the  company  must  work 
closely  with  him  or  her  to  ensure  the  most  important  items  are  evaluated.  The  questions  should 
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be  designed  to  retrieve  and  provide  both  numerical  and  descriptive  data.  Actual  times  of  service 
and  the  number  of  telephone  rings  are  numerical  data,  whereas  the  degree  of  staff  friendliness  or 
feel  of  ambiance  is  descriptive  data.  Even  though  scaled  ratings  are  needed  for  tracking  trends, 
written  comments  enhance  the  quality  of  feedback.  According  to  Ann  Jennings,  founder  of 
T.I.P.S.  shopping  service,  “the  response  is  only  as  good  as  the  question,  so  write  the  questions  to 
get  the  right  information.  This  information  is  used  to  make  business  decisions”  (Thompson, 
1993,  p.  27). 

Program  Costs 

There  is  a  wide  range  of  costs  associated  with  mystery  shopping.  The  average  cost  per 
shop  is  difficult  to  pinpoint  because  each  company’s  requirements  are  different.  Complexity  of 
feedback,  in-house  vs.  contract,  volume  of  shops,  experience  of  shoppers,  and  price  of  the 
services  will  effect  shopping  prices.  As  a  minimum,  most  companies  cover  the  price  of  the  meal 
or  hotel  stay  up  to  a  set  amount. 

In-house  programs,  for  example,  have  costs  associated  with  recruiting,  training, 
questionnaire  design,  scheduling,  and  of  course  the  cost  of  services  and  products  consumed  plus 
additional  pay  for  the  shopper.  When  Au  Bon  Pain  ran  their  in-house  program,  they  estimated  it 
cost  more  than  $150,000,  including  internal  administration.  Au  Bon  Pain  paid  in-house  shoppers 
$10  a  visit,  and  the  contract  shopping  company  $30  a  visit  inclusive  of  food  (Brokaw,  1991). 
Frequently,  at  more  expensive  restaurants  and  hotels,  free  food  or  accommodations  is  adequate 
compensation  for  the  shoppers  (Jedd,  1994). 

Independent  shopping  companies  are  typically  paid  $50  per  visit  plus  expenses  for  a 
restaurant  visit,  although  larger  contracts  with  more  frequent  shops  can  receive  a  discount  price 
for  volume  (Kelly,  1997).  Hotel  shops  are  more  expensive.  The  cost  of  a  hotel  shop  can  go  as 


high  as  $1,500  plus  expenses  for  a  shopper  staying  overnight  at  a  full-service  resort  hotel  (Ford 
and  Bach,  1997). 
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Frequency  of  Shops 

The  frequency  of  visits  will  depend  on  what  is  being  evaluated.  Most  companies  have 
their  operations  shopped  once  a  month  (Brokaw,  1991;  Brown,  1989;  Quinton,  1991).  If  data  is 
needed  on  different  shifts,  each  one  should  be  shopped  monthly.  Less  frequent  shops  reduce  the 
validity  of  the  data.  The  time  when  shops  occur  is  important  as  well.  “A  spotting  at  peak  time 
will  be  more  useful  than  a  spotting  at  off  peak,  especially  if  a  competitive  comparison  is  being 
made”  (Kelly,  1997,  p.  18).  Max  and  Erma’s  Restaurants,  for  example,  are  shopped  a  minimum 
of  16  times  each  per  year  (Quinton,  1991).  Ruby’s  restaurant  chain  shops  their  stores  once  per 
month  and  Au  Bon  Pain  has  shoppers  visit  each  store  three  times  over  a  four  week  period  to 
check  each  shift  (Brokaw,  1991;  Brown,  1989). 

Recognition  Programs 

“Managers  and  supervisors  should  always  focus  on  the  positive;  emphasizing  what  could 
have  been  done  more  effectively,  as  opposed  to  what  was  done  wrong”  (Kennedy,  1997,  p.  37). 
Karen  Dorman,  President  of  ServiceSeekers  (1994)  agrees  stating  “one  of  the  most  effective  uses 
of  shopping  results  is  a  comprehensive  incentive  program.  This  is  best  accomplished  by  using 
positive  survey  results  to  reward  employees  for  approprzdte  behaviors  and  to  retrain,  not 
condemn,  employees  for  deficiencies  the  survey  detected”  (p.  18).  The  important  part  of  this  is 
recognizing  the  employees  that  provided  outstanding  service.  Employees  will  have  a  better  view 
of  the  program  when  it  rewards  them  for  their  hard  work  than  trying  to  find  them  making  a 
mistake.  David  Lipton,  President  of  Sensors  Quality  Management  Inc.  believes  that  mystery 


shopping  “is  a  tool  to  change  or  develop  employee  behaviors.  Incentives,  he  says,  are  more 
powerful  than  Big  Brother”  (Cook,  1998). 
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Most  companies  have  a  recognition  program  tied  into  their  mystery  shopper  program. 
“Shoney’s,  Max  and  Erma’s,  Au  Bon  Pain  and  many  other  restaurants  base  incentive  programs 
and  bonuses  on  mystery  shopper’s  reports”  (Thompson,  1993,  p.  26).  Ruby’s  owners  reward 
waiters  who  score  high  on  the  shop  with  $50  as  soon  as  the  survey  is  turned  in,  and  manager 
bonuses  are  also  influenced  by  the  shopping  reports.  Restaurants  Unlimited  in  Seattle  links 
shopping  reports  to  a  system  of  incentives  for  salaried  employees  as  well  as  bonuses  for 
management  (Quinton,  1991).  “There  are  few  better  ways  of  influencing  employees  about  your 
commitment  to  providing  excellent  service  than  by  making  an  example  of  those  employees  who 
do”  (Kelly,  1997,  p.  16). 

Training 

There  is  very  little  information  in  the  literature  concerning  training  for  mystery  shoppers. 
Steven  Snedegar,  President  of  Guest  Perception  Inc.,  explains  that  companies  cannot  train 
independent  mystery  shoppers  in  the  same  way  that  they  train  their  own  employees,  otherwise 
the  shoppers  could  be  viewed  by  the  IRS  as  employees  instead  of  independent  subcontractors 
(1998).  “In  order  to  be  classified  as  subcontractors,  they  (shoppers)  need  to  receive  their  training 
from  experience,  or  through  other  industry  or  group  literature”  (Snedegar,  1998,  p.3).  This 
indicates  that  shoppers  do  not  normally  receive  training  unless  they  are  an  employee  of  the 
company.  If  this  is  the  case,  then  many  shops  are  often  completed  by  untrained  beginner 
shoppers.  Au  Bon  Pain  appears  to  be  an  exception.  They  provide  each  shopper  with  a  2-3 
training  class  to  teach  them  about  the  company’s  service  philosophy  and  shopping  program 


(Brokaw,  1991). 
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Summary 

Service  quality  is  as  important  to  Air  Force  Services  as  it  is  in  commercial  hotel  and 
restaurant  companies.  Mystery  shopping  is  widely  used  commercially  as  a  tool  to  identify 
service  quality  problem  areas  and  measure  performance.  Industry  standards  for  frequency,  cost, 
recognition,  and  training  can  be  used  as  guides  for  developing  an  Air  Force  mystery  shopping 


program. 
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CHAPTER  III 

DESIGN  AND  METHODOLOGY 

This  project  was  completed  in  two  parts.  The  first  was  a  telephone  survey  of  commercial 
chains  and  independent  mystery  shopping  companies  to  determine  the  industry  standards  for 
shopping  frequency,  cost  per  shop,  and  recognition  programs.  The  second  was  the  design  and 
testing  of  a  mystery  shopping  program  materials  for  the  Wright-Patterson  AFB  Services 
Squadron  based  on  the  industry  standards  as  applicable. 

Telephone  Survey  of  Corporate  and  Independent  Mystery  Shopping  Companies 

The  survey  was  conducted  by  telephone  of  5  hotel,  5  restaurant,  and  5  independent 
shopper  companies  for  a  total  of  15  survey  participants.  Separate  surveys  were  designed  for  the 
shopping  companies  and  the  corporate  chains  (See  Appendix  A). 

The  corporate  survey  was  used  to  identify  if  the  company  used  an  internal  shopping 
program  or  hired  an  external  contractor.  The  survey  also  looked  at  training,  questionnaire 
design,  areas  of  evaluation,  frequency,  costs,  recruiting,  recognition,  and  problems  most  often 
encountered  with  the  shopping  program.  Finally,  companies  were  requested  to  send  a  sample 
copy  of  their  questionnaire.  The  shopping  company  survey  was  similar.  It  asked  questions 
about  training,  questionnaire  design,  areas  of  evaluation,  frequency,  costs,  shopper 
compensation,  recruiting,  and  problems  most  often  encountered  with  the  shopping  program. 
Although  many  of  the  areas  evaluated  on  each  form  were  similar,  the  questions  were  worded 
differently  to  properly  fit  each  situation.  To  ask  about  costs  for  example,  the  corporate  company 
was  asked  “How  much  do  you  estimate  each  shop  cost?”  while  the  shopper  company  was  asked 
“On  average  how  much  do  you  charge  per  shop  above  the  cost  of  the  food/services  utilized?” 
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The  companies  were  not  randomly  selected  for  the  survey.  Hotel  and  restaurant 
companies  were  chosen  based  their  having  characteristics  similar  to  Air  Force  lodging  and 
foodservice.  Large  hotel  chains  in  the  mid-priced  range  and  restaurants  with  casual  dining 
themes  or  buffets  were  selected.  Companies  that  were  cited  in  articles  as  having  a  mystery 
shopping  program  that  fit  the  above  criteria  were  selected  to  improve  response  on  mystery 
shopping  standards. 

The  calls  were  conducted  over  a  two- week  period  in  June  1998.  Phone  numbers 
were  obtained  from  hotel  and  restaurant  directories.  Many  companies  did  not  have  mystery 
shopping  programs,  did  not  wish  to  discuss  their  program  with  me,  or  were  not  assessable  after 
numerous  attempts  (person  on  vacation,  not  returning  calls,  busy  signal,  no  answer,  etc.). 
Companies  were  called  until  at  least  five  participants  were  obtained  in  each  of  the  three 
categories  listed  above. 

The  restaurant  companies  that  responded  to  the  survey  were  Bob  Evans  Restaurants, 
Applebee’s  International,  Au  Bon  Pain  Co.  Inc.,  Old  Country  Buffets,  and  Shoney’s  Inc.  The 
hotel  companies  that  responded  to  the  survey  were  Marriott  Residence  Inn,  Radisson  Hotels 
Worldwide,  Hyatt  Hotels  Corp.,  Bass  Hotels  and  Resorts,  and  Promus  Hotel  Corp..  The 
independent  mystery  shopping  companies  that  responded  to  the  survey  were  Feedback  Plus, 
Guest  Perception  fric.,  BMA,  Richey  International,  and  A  Closer  Look. 

Design  and  Testing  of  the  Mystery  Shopping  Program 

The  second  part  of  the  project  was  the  design  and  testing  of  the  mystery  shopping 
program  at  Wright-Patterson  Air  Force  Base.  After  meeting  with  the  marketing  and  training 
managers  for  the  Services  Division  on  base,  it  was  decided  the  main  program  elements  to  be 
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developed  were:  marketing  materials,  recruiting  materials,  evaluation  forms,  training  program, 
shopping  schedule,  program  budget,  recognition  program,  results  analysis  database  and  other 
administrative  requirements  (see  Appendix  B).  A  timeline  for  the  program  was  also  drawn  up  to 
identify  when  each  element  needed  to  be  completed  (see  Appendix  C). 

Marketing  Materials 

To  help  promote  the  program  and  give  it  an  identity,  a  logo  was  designed 
for  the  program  (see  Figure  3.1).  This  logo  was  placed  on  all  recruiting  and 
promotional  materials  for  the  program.  It  was  agreed  that  a  standard  logo  would 
help  employees  and  customers  easily  recognize  and  remember  the  program.  In  addition,  a  slogan 
was  developed  called  “Caught  you  doin’  it  Wright!”  to  catch  the  employees  eyes  and  create  a 
positive  feeling  about  the  purpose  of  the  program.  The  phrase  plays  on  the  word  “right”  by  using 
“Wright.”  Wright  is  the  namesake  for  this  Air  Force  base  named  after  Orville  and  Wilbur 
Wright  who  designed  the  first  successful  heavier-than-air  aircraft. 

Recruiting  Shoppers 

The  shoppers  were  recruited  from  the  general  base  population,  which  numbered  around 
35,000.  Since  the  population  was  represented  by  distinct  groups  to  include,  officers,  enlisted, 
reservists,  retirees,  government  civilians,  military  family  members  and  government  contractors,  a 
broad-based  recraiting  program  was  used.  Tri-fold  informational  recmiting  brochures  we  made 
to  entice  potential  customers  to  become  mystery  shoppers  (see  Appendix  D).  The  brochure 
highlighted  the  logo  and  gave  basic  information  about  the  program.  It  also  provided  contact 
information  and  a  mail-in  form  that  prospective  shoppers  could  complete.  These  recraiting 
brochures  were  distributed  at  the  biweekly  Newcomer’s  Orientation,  the  Retiree  Affairs  Office, 
the  reserve  Public  Affairs  Office,  and  placed  in  the  lodging  lobby  and  at  each  foodservice 


Figure  3.1 
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facility.  In  addition,  advertisements  were  placed  in  the  electronic  Weekly  Bulletin  and  in  the 
base  newspaper.  These  recruiting  methods  ensured  that  a  large  percentage  of  the  base  population 
including  members  of  each  demographic  category  would  be  notified  of  the  program. 

Budget  Development  and  Program  Costs 

A  comprehensive  budget  was  developed  to  estimate  the  costs  of  the  program  (see 
Appendix  E).  This  document,  written  in  Microsoft  Excel  97  (hereafter  “Excel”),  tracked  the 
activities  to  be  shopped,  the  frequency  of  shops,  the  budgeted  cost  per  shop,  and  the  actual 
payment  to  shoppers.  This  information  was  totaled  by  flight  and  overall  division.  The  budget 
automatically  updated  the  actual  expenditures  when  each  form  was  entered  into  the  mystery 
shopper  database.  The  costs  of  the  program  included  reimbursement  of  the  shopping  expenses 
(up  to  a  set  amount),  gift  certificates  for  the  shoppers,  labor  expenses  to  manage  the  program, 
marketing  materials,  and  employee  recognition. 

Available  funding  and  nuances  in  Air  Force  monetary  regulations  shaped  the  shopper 
payment  procedures.  Upon  completing  the  shop  each  shopper  submitted  his  or  her  completed 
form  and  receipt.  The  receipt  was  validated  and  the  shopper  was  provided  reimbursement  up  to 
the  maximum  amount  set  for  the  shop.  The  reimbursement  limit  ranged  from  $5.00  to  $20.00 
based  on  the  activity  being  shopped  and  the  meal  being  shopped.  The  shopper  was  then 
presented  with  two  $5.(X)  gift  certificates  to  use  in  any  Services  activity.  The  gift  certificates 
could  not  be  used  in  conjunction  with  another  shop,  other  discounts  or  promotions,  or  to 
purchase  alcohol.  No  change  was  provided  if  the  full  value  of  the  certificate  was  not  used. 

These  procedures  were  explained  to  the  shoppers  in  the  Guide  to  Mystery  Shopping  and  in  the 
training  classes. 


The  shoppers  were  hired  as  “independent  contractors”  for  each  shop.  The  informal 
contractual  agreement  was  ended  upon  completion  of  the  shop  and  payment  to  the  shopper. 
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There  was  no  guarantee  for  any  future  shopping  opportunities,  nor  was  there  a  minimum  number 
of  shops  that  a  shopper  had  to  complete.  Shoppers  could  remove  their  name  from  the  program  at 
any  time. 

Frequency  of  Shops 

When  initially  determining  the  frequency  of  shops  for  each  activity,  one  shop  per  month 
was  recommended  based  on  the  industry  standard.  However,  this  number  was  modified  by  the 
Director  of  Services  based  on  his  desire  to  obtain  more  data  on  certain  activities  than  on  others. 

It  was  also  effected  by  monetary  constraints.  Since  there  were  three  different  types  of  base 
restaurant  formats  (table  dining  restaurants,  cafeteria,  and  small  canteen)  the  annual  shopping 
frequency  was  set  differently  for  each  format.  The  two  table  dining  activities  were  budgeted  for 
a  total  of  twelve  shops.  The  three  cafeterias  combined  were  also  budgeted  for  a  total  of  twelve 
shops.  The  twelve  canteens  combined  were  budgeted  for  a  total  of  twenty-four  shops.  The  golf 
and  bowling  activities  were  set  at  six  annual  shops  each  because  they  are  seasonal  activities.  The 
golf  courses  will  normally  be  shopped  in  the  summer  months  while  the  bowling  center  will  be 
shopped  primarily  in  the  winter  months.  The  Officers’  Club,  the  Flywright  Club  and  lodging 
were  budgeted  for  one  shop  per  month.  For  this  project,  the  frequency  was  accelerated  to  obtain 
enough  shops  to  evaluate  the  program. 

Shopping  Schedule 

A  shopping  schedule  was  developed  to  identify  the  exact  place  and  time  for  each 
budgeted  shop  (see  Appendix  F).  This  schedule  tracked  whether  each  shop  had  been  completed 
or  not.  This  schedule  also  tracked  what  activities  had  been  shopped  and  what  needed  to  be 
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shopped.  It  allowed  the  user  to  spread  the  shops  out  so  that  they  did  not  occur  at  the  same 
location  on  the  same  day.  The  shopping  schedule  aided  the  program  manager  in  setting  the 
entire  year’s  shopping  schedule  in  advance.  The  annual  shopping  schedule  will  not  be 
completed  until  the  Services  Division  has  reviewed  the  results  of  the  initial  program  test  and 
implemented  necessary  changes. 

Development  of  Evaluation  Forms 

The  most  important  areas  to  be  evaluated  in  all  facilities  were  identified  before  the 
evaluation  forms  were  developed.  Input  was  obtained  from  Services’  customers  using  a  1998 
customer  satisfaction  survey.  The  survey,  which  received  694  responses,  identified  the  most 
important  attributes  of  Services’  activities  as  ranked  by  the  customers  (See  Appendix  G).  The 
ten  highest  attributes  were;  employee  courtesy,  service  correctness,  facility  cleanliness, 
employee  helpfiilness,  equipment  condition,  equipment  cleanliness,  service  promptness,  product 
knowledge,  caring  service,  and  understanding  customer  needs. 

Input  was  also  obtained  from  management  and  the  training  director.  Each  manager 
submitted  a  list  of  the  most  important  areas  they  wanted  the  mystery  shopper  to  evaluate.  The 
training  director  submitted  questions  that  reflected  the  materials  every  employee  had  been 
trained  on.  Input  from  the  training  manger  helped  devise  questions  to  validate  the  training  that 
had  been  provided.  From  these  three  sources,  evaluation  forms  were  developed  using  Excel  for 
lodging  and  each  type  of  foodservice  facility  (see  Appendix  H  for  examples). 

The  foodservice  evaluation  forms  were  divided  into  the  following  categories:  base 
restaurants  -  table  dining,  base  restaurants  -  cafeterias,  base  restaurants  -  canteens.  Officers’ 
Club  -  table  dining,  Officer’s  Club  -  buffet  dining,  military  dining  facility,  golf  courses,  and 
bowling  center. 
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Questions  were  developed  to  rate  common  areas  in  all  facilities  in  the  Services  Division. 
Twenty-one  questions  were  identified  as  common  among  every  activity  (see  Appendix  I).  The 
rest  of  the  questions  were  designed  to  be  specific  to  the  type  of  activity  being  evaluated.  To 
prevent  the  forms  from  being  too  long,  the  maximum  length  was  set  at  two  pages.  Most  of  the 
questionnaires  were  only  I-V2  pages  in  length. 

The  evaluation  forms’  questions  were  written  as  either  an  objective  Yes/No  question,  or 
as  a  subjective  rated  question.  The  Yes/No  questions  were  designed  to  obtain  a  “Yes”  response 
if  the  activity  met  the  standard.  Any  “No”  response  indicated  there  was  a  problem.  The  rated 
questions  were  based  on  a  scale  from  1  to  5,  with  1  being  the  lowest  and  5  being  the  highest. 

This  scale  was  determined  by  the  Marketing  Department  to  match  the  annual  customer  survey, 
on-going  activity  surveys,  and  the  Division’s  customer  comment  card.  This  allowed  the  mystery 
shopping  data  to  be  compared  on  a  similar  scale  with  the  other  feedback  methods  used. 

Shoppers  were  trained  to  use  only  whole  numbers  when  using  the  rating  scale.  The  Division  also 
decided  that  the  “5”  rating  should  not  be  used  only  for  perfection,  but  to  recognize  “outstanding” 
and  “the  way  it  should  be.”  This  nuance  to  the  rating  scale  was  emphasized  to  increase  the 
number  of  awards  presented  to  Division  employees. 

Written  Shopping  Instructions 

Written  shopping  instructions  were  developed  for  each  activity  to  be  shopped  (see 
Appendix  J  for  examples).  All  of  the  instructions  were  one  page  in  length  and  divided  into  three 
sections.  The  first  section  told  the  shopper  where  to  complete  the  shop,  when  to  complete  the 
shop  and  how  much  would  be  reimbursed.  The  middle  section  was  standardized  on  all 
instruction  sheets  and  included  important  reminders  about  completing  the  shop.  For  example, 
the  5-point  rating  scale,  obtaining  a  receipt,  and  writing  comments  for  all  low  ratings  were 
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reiterated.  The  last  section  was  specific  to  that  activity.  It  provided  the  shopper  with  the  phone 
number  to  call  for  the  pre-visit  phone  call.  It  also  gave  directions  about  what  should  be  shopped 
(i.e.  visit  the  restroom  and  look  at  the  salad  bar).  This  section  was  intended  to  focus  the  shopper 
on  what  was  important  to  do  for  the  shop. 

The  instruction  sheets  were  designed  based  on  commercially  used  instructions  and  input 
from  the  managers  on  what  the  shopper  should  know  right  before  completing  the  shop.  In  the 
training  sessions,  the  shoppers  were  told  to  review  the  instructions  and  forms  right  before 
entering  the  facility  to  complete  the  shop. 

The  lodging  instructions  were  different  than  the  rest  (see  Appendix  J).  Because  of  the 
unique  nature  of  Air  Force  Lodging,  the  shopper  needed  to  follow  specific  instmctions  that 
would  enable  them  to  obtain  a  reservation  and  a  room  in  lodging.  As  a  result,  these  instructions 
were  more  complicated  than  the  other  facilities.  The  lodging  manager  was  aware  of  the  presence 
of  a  shopper  in  advance  because  his  involvement  was  the  only  way  a  shopper  could  successfully 
obtain  a  room.  After  the  lodging  shop  was  completed,  the  shopper  was  required  to  contact  the 
Executive  Housekeeper  to  obtain  the  name  of  the  housekeeper  for  their  room. 

Development  of  Training  Materials 

A  detailed  manual  called  the  88*  Services’  Guide  to  Mystery  Shopping,  also  referred  to  a 
the  Guide  (see  Appendix  K),  was  developed  as  a  training  aid  and  take  home  reference  for 
shoppers  to  use.  This  guide  outlined  the  purpose  of  the  program,  advice  on  how  to  complete  a 
shop  (including  sample  shops),  and  administrative  details  for  turning  in  shops  and  receiving 
reimbursement.  The  guide  provided  essential  information  for  the  shopper  to  reference  at  any 
time  and  refreshed  the  shopper’s  memory  of  the  procedures  before  each  shop. 
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In  addition,  training  classes  were  designed  and  presented  to  perspective  mystery  shoppers 
for  several  purposes.  The  first  was  to  explain  the  program  and  motivate  the  shoppers  to  join.  The 
second  was  to  prepare  the  shoppers  in  advance  how  to  shop  successfully  and,  the  last  was  to 
explain  the  administrative  process  and  emphasize  key  policies  and  procedures.  These  classes 
were  held  in  a  small  training  room  centrally  located  on  base.  A  maximum  of  fifteen  students  was 
allowed  in  each  class  to  promote  a  more  intimate  and  casual  atmosphere  and  to  encourage  group 
discussion  and  questions.  Several  one-on-one  sessions  were  conducted  to  train  shoppers  who 
were  unable  to  make  the  training  session.  Four  regular  sessions  and  four  one-on-one  training 
sessions  were  conducted.  In  all,  32  shoppers  were  trained  for  the  program.  Most  of  these 
shoppers  were  assigned  shops  immediately  following  their  training.  The  Services  Training 
Manager  was  an  integral  part  of  these  training  sessions  by  conducting  most  of  the  set-up  and 
instruction. 

Shopper  Demographic  Information 

. 

At  each  training  session  the  shoppers  were  asked  to  complete  a  demographic 
questionnaire  (see  Appendix  L).  This  questionnaire  was  designed  to  obtain  demographic 

I 

information  about  the  shoppers.  The  questionnaire  collected  information  about  military  category 
(i.e.  officer,  enlisted,  retiree,  etc.),  age,  gender,  family  size,  marital  status,  contact  information  (e- 
mail,  address,  etc.)  and  what  Services  activities  they  normally  use  or  are  interested  in  using. 
Several  of  the  activities  such  as  the  Officers’  Club  require  membership,  so  this  was  also  a 
question  on  the  questionnaire.  The  questionnaire  was  explained  in  detail  during  the  training 
sessions.  Shops  were  assigned  to  shoppers  who  closely  matched  the  demographic  profile  of  the 
activity’s  regular  customers. 

j 

j 

I 
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The  information  on  each  shopper  was  entered  into  an  Excel  spreadsheet  (see  Appendix 
M)  to  allow  sorting  and  querying  of  the  data.  An  e-mail  address  list  was  established  in  the 
Services  Marketing  Department  to  allow  quick  and  easy  flow  of  information  to  the  shoppers. 
Feedback  from  the  first  wave  of  completed  shops  was  sent  out  to  all  of  the  shoppers  to  improve 
results  on  the  later  shops.  This  was  also  used  to  advertise  upcoming  shops. 

Recognition  Program 

A  recognition  program  was  set  up  to  reward  employees  for  providing  outstanding 
customer  service  and  encourage  standards  were  maintained.  An  important  consideration  in  the 
determination  of  the  recognition  program  was  to  provide  equal  awards  to  all  three  categories  of 
Services’  employees  (military,  appropriated  fund  civilians,  and  non-appropriated  fund  civilians). 
As  a  result,  employees  rated  for  providing  “outstanding”  service  (rated  a  “5”  on  the  evaluation 
form)  were  recognized  with  a  certificate,  a  lapel  pin  (see  Appendix  N),  and  one  paid  hour  off. 
The  lapel  pin  and  certificate  highlight  the  slogan  of  the  program  “Caught  doin’  it  Wright!” 
Employees  were  informed  about  the  mystery  shopping  program  through  supervisors  and  with 
large  posters  advertising  the  program  (see  Appendix  O).  These  posters  showed  examples  of  the 
form  that  was  used  to  shop  their  activity  and  highlighted  the  recognition  program. 

Shoppers  were  trained  to  obtain  the  first  name  and  description  (unless  it  was  a  pre- visit 
phone  call)  of  the  employees  who  provided  them  service.  This  was  strongly  emphasized  in  the 
training  sessions  and  the  Guide  to  Mystery  Shopping.  The  name,  description,  overall  rating,  and 
shop  location  information  was  automatically  transferred  to  the  recognition  database  when  the 
shop  was  entered  in  Excel.  The  mystery  shopper  program  manager  then  used  this  information  to 
identify  the  full  name  of  the  employee  and  update  the  database.  A  recognition  certificate  was 
printed  for  all  employees  rated  as  a  “5.”  Employees  were  only  recognized  once  per  shop,  even  if 
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they  were  rated  a  “5”  on  two  different  areas  of  the  evaluation  form.  Employees  who  were 
recognized  during  two  or  more  shops  did  not  receive  additional  pins,  only  the  certificate  and  time 
off. 

Database  Development 

An  Excel  database  was  developed  to  collect  results  from  each  completed  evaluation  form. 
The  design  and  importance  of  the  database  grew  as  the  mystery  shopper  program  progressed.  It 
was  discovered  that  many  of  the  functions  needed  to  track  the  results  of  the  program  could  be 
automated  in  Excel.  As  a  result,  all  possible  aspects  of  the  program  were  integrated  into  one 
Excel  program  called  “Shoppy!”  These  elements  consisted  of  the  evaluation  forms,  budget, 
shopper  schedule,  recognition  database,  results  database,  and  shopper  database.  The  program 
was  fully  controlled  by  a  Microsoft  Visual  Basic  driven  menu  and  control  program.  This  control 
program  allowed  for  full  integration  of  the  different  mystery  shopping  elements  above. 

The  program  centered  on  the  evaluation  forms  that  allowed  the  forms  to  be  printed  and 
the  completed  shopping  data  to  be  entered  into  the  program.  From  the  forms,  the  entered  data 
was  automatically  transferred  to  either  the  recognition  database,  the  budget,  or  to  the  results 
database.  The  results  database  collected  the  responses  from  the  twenty-one  commonly  rated 
questions  on  all  forms  (see  Appendix  P).  This  database  could  then  be  manipulated  to  provide 
statistical  summaries,  graphs  and  other  information  the  Services  Division  desired. 

Program  Flexibility 

The  entire  mystery  shopping  program  was  designed  to  be  flexible  to  allow  for  expansion 
to  other  activities  in  the  Division  and  for  other  Air  Force  bases  to  use.  As  a  result,  generic  names 
of  activities  were  used  instead  of  formal  names  wherever  possible.  For  example,  the  BQttyhawk 
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Bowling  Center  was  referred  to  as  just  Bowling  Center  in  almost  all  situations.  This  will  enable 
other  bases  with  similar  activities  to  use  the  system. 

The  Visual  Basic  program  was  also  designed  generically  to  allow  expansion  to  the  rest  of 
the  Division’s  activities  such  as  the  auto  skills  center,  fitness  centers,  and  outdoor  recreation. 
There  are  few  formal  names  used  within  the  programming  code.  The  budget  spreadsheet  was 
also  written  to  include  the  entire  Division. 

Program  Critique 

To  discover  the  effectiveness  of  the  program  from  the  shopper’s  perspective,  a  critique 
form  was  developed  and  attached  to  every  shop  (see  Appendix  Q).  This  critique  asked  the 
shopper  about  the  effectiveness  of  the  training  information,  training  techniques,  evaluation 
forms,  shopping  instructions,  and  the  Guide  to  Mystery  Shopping.  In  addition,  shoppers  were 
asked  to  provide  suggestions  to  make  each  of  these  aspects  better.  Finally,  shoppers  were  asked 
if  this  program  positively  affected  their  opinion  of  the  Services  Division. 

The  shoppers  were  selected  to  provide  feedback  because  they  could  provide  an  outside 
perspective  concerning  the  effectiveness  of  the  program.  Feedback  surveys  of  employees  and 
managers  were  considered,  but  the  lengthy  Air  Force  approval  process  and  relatively  short 
timeline  of  this  project  precluded  this  option.  The  critique  was  explained  to  the  shoppers  during 
the  training  sessions.  Shoppers  were  requested  to  complete  a  critique  for  every  shop  they 
performed  because  the  different  shops  used  different  forms  and  instructions. 

Testing  of  the  Program 

To  test  the  mystery  shopping  program,  10  lodging  shops  and  40  foodservice  shops  were 
scheduled.  The  shops  were  assigned  to  the  trained  shoppers  between  August  19*’’  and  October 
27*,  1993  either  at  the  training  session,  by  telephone,  or  by  e-mail.  The  Services  Marketing 
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Department  collected  the  completed  shops.  Several  adjustments  were  made  to  the  shopping 
schedule  in  order  to  complete  the  large  number  of  shops  during  the  short  period  of  time. 
Shoppers  were  selected  for  the  shops  based  on  their  demographic  profile.  For  example,  the 
primary  customers  of  lodging  are  officers,  enlisted  and  retirees.  There  were  very  few  shoppers 
in  these  categories  that  were  willing  to  shop  lodging  overnight.  As  a  result  only  seven  of  the  ten 
scheduled  shops  were  completed  on  lodging  to  prevent  from  having  too  many  shops  that  did  not 
match  the  primary  customer  profile.  The  results  obtained  from  the  shops  were  keyed  into  the 
mystery  shopper  database  program  to  test  the  program  with  real  data. 
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CHAPTER  IV 

RESULTS  AND  DATA  ANALYSIS 

Telephone  Survey  Results 

The  telephone  survey  collected  information  from  five  hotel,  restaurant  and  independent 
mystery  shopper  companies  (referred  to  hereafter  as  “consulting  companies”).  The  survey 
collected  information  on  internal  vs.  external  programs,  shopper  recruiting,  training  of  shoppers, 
evaluation  areas,  frequency  of  shops,  cost  of  shops,  recognition  programs  and  problem  areas  (see 
Appendix  R  for  detailed  responses).  There  was  a  large  variance  between  the  types  of  programs 
being  used  by  the  different  companies.  As  a  result,  several  of  the  questions  only  obtained 
responses  from  a  portion  of  the  respondents.  Each  area  was  summarized  to  identify  common 
trends. 

Internal  vs.  contract  mystery  shopping  program.  The  hotel  companies  all  used  internal 
mystery  shopping  programs.  Only  Marriott’s  Residence  Inn  called  their  program  “mystery 
shopping.”  Bass,  Promus,  and  Radisson  used  the  term  quality  assurance  (QA).  Since  these  QA 
programs  conducted  unannounced  evaluations  on  their  properties  and  had  a  similar  structure, 
they  were  categorized  as  a  form  of  mystery  shopping  for  this  survey.  Hyatt  did  not  have  a 
program  that  completed  unannounced  evaluations  on  their  properties,  although,  it  was  likely  that 
mystery  shopping  was  used  on  a  decentralized  basis  within  the  chain.  Instead  they  used  a  guest 
call-back  program  to  obtain  the  customer’s  perspective  on  their  product. 

The  restaurant  companies  predominately  used  more  conunercial  mystery  shopping 
companies.  Bob  Evans  had  a  very  active  externally  mn  program.  Applebee’s  used  an  external 
company  to  run  a  variant  of  mystery  shopping  called  the  customer  service  index  (CSI)  program 
which  obtains  customer  feedback  through  a  1-800  number.  This  will  be  discussed  more  in  the 
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Other  Comments  and  Unexpected  Findings  section  below.  An  Bob  Pain  disbanded  their 
program  and  only  used  a  small  quality  assurance  program.  Shoney’s  had  an  internally  run 
program.  Old  Country  Buffets  and  Luby’s  used  external  programs.  Although  Luby’s  refused  to 
complete  the  survey,  they  did  acknowledge  that  they  used  mystery  shopping  and  hired  an 
external  contractor.  Overall,  the  restaurants  hired  mostly  external  contracted  companies  to 
complete  their  mystery  shopping.  The  “internal  vs.  external  program”  question  did  not  apply  to 
the  consulting  companies. 

Recruiting  Shoppers.  All  of  the  hotel  companies  used  full  time  employees  to  conduct 
their  shopping.  Marriott  also  contracted  with  five  individuals  to  help  complete  their  shops. 

The  restaurant  companies  that  used  an  external  contractor  relied  on  the  contractor  to 
recruit  the  shoppers  except  for  Applebee’s,  which  used  their  POS  system  to  select  every  30* 
customer.  Shoney’s  recruited  customers  from  business  cards  left  at  the  restaurants  and  through 
their  web  site.  Au  Bon  Pain  used  their  managers  to  perform  the  shops. 

The  consulting  companies  stated  that  referrals  were  the  best  method  of  recruiting.  Three 
companies  used  the  Internet  to  recruit  through  their  web  sites.  As  a  last  resort,  consulting 
companies  placed  newspaper  ads,  cold-called  potential  shoppers  at  their  workplace,  or  sent 
advertisements  in  the  mail.  Richey  International  was  the  only  consulting  company  surveyed  to 
hire  all  full-time  shoppers. 

Training  shoppers.  For  the  most  part,  hotel  companies  provided  specialized  training  for 
their  evaluators.  These  programs  included  shadowing  other  evaluators,  listening  to  customer 
calls,  and  visiting  different  properties.  Since  most  of  the  restaurant  companies  used  a  consulting 
contractor,  they  did  not  train  the  shoppers.  Shoney’s  sent  detailed  instructions  through  the  mail 
with  the  evaluation  forms.  Au  Bon  Pain  used  managers  but  did  not  provide  them  with 


specialized  evaluation  training.  Bob  Evans  required  their  contractor  to  hire  experienced 
shoppers  and  train  them  specifically  for  Bob  Evan’s  shops. 


33 


Consulting  companies  often  relied  on  the  shoppers  having  previous  experience.  In 
addition,  they  usually  sent  detailed  instructions  with  the  evaluation  forms  and  provided  shoppers 
feedback  after  the  shop  was  performed  to  help  them  with  the  next  shop.  Feedback  Plus  offered  a 
1-800  number  and  encouraged  shoppers  to  ask  questions  and  obtain  clarification  before  they 
shopped.  Richey  International  had  a  four  to  five  month  training  program,  which  consisted  of 
spending  one  week  at  a  non-client  hotel  property  and  then  three  to  four  months  as  an  extra  body 
on  a  regular  evaluation  team. 

Questionnaires  tailored  to  company  being  shopped.  Across  the  board  all  hotel, 
restaurant,  and  consulting  companies  tailored  their  evaluation  forms  to  the  specific  company 
being  shopped.  Richey  International  indicated  that  they  also  used  some  standard  forms  as  well. 
Many  of  the  companies,  especially  the  consulting  companies,  stated  that  the  questionnaire  should 
consist  mostly  of  objective  questions  because  they  were  easier  to  measure.  Applebee’s  and 
Hyatt,  which  had  similar  programs,  used  a  14-question  survey  to  obtain  customer  feedback. 

Areas  of  evaluation.  Almost  every  survey  participant  identified  service  quality  as  an 
important  area  to  evaluate.  Cleanliness  was  another  commonly  evaluated  area.  The  most 
important  areas  of  evaluation  varied  for  the  consulting  companies  because  the  priorities  of  each 
client  varied  from  the  next.  The  consulting  companies  also  identified  employee  greeting, 
nametags,  and  clean  restrooms  as  important  areas.  Very  few  respondents  mentioned  product 
quality  as  a  high  priority.  Marriott  only  shopped  by  telephone  checking  reservations  and  general 


information. 
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Frequency.  The  hotel  companies  that  conducted  full  overnight  evaluations  shopped  each 
property  once  or  twice  a  year.  Properties  that  had  more  problems  were  evaluated  an  additional 
one  to  two  times.  Marriott  called  each  property  once  per  week  during  different  shifts.  Hyatt 
called  80,000  customers  with  an  80%  response  rate  during  the  year. 

The  restaurant  companies  had  a  higher  frequency  than  the  hotels.  Most  companies 
shopped  four  times  per  month.  Two  of  the  companies.  Bob  Evans  and  Old  Country  Buffets  have 
reduced  their  frequency  to  around  two  shops  per  month.  These  companies  felt  they  had  made 
significant  improvements  with  the  four  shops  per  month,  and  felt  that  they  could  maintain  then- 
program  with  fewer  shops.  Applebee’s  selects  every  30*  customer  for  a  shop.  Last  year  the 
company  obtained  300,000  responses  to  their  customer  call-in  program. 

The  consulting  companies  recommended  that  restaurants  be  shopped  from  one  to  four 
times  per  month,  more  for  trouble  properties.  The  hotel  frequency  varied  more.  They 
recommended  that  hotels  be  shopped  from  once  per  year  to  four  times  per  month.  Richey 
International  which  shops  mostly  hotels,  reconunended  that  a  property  should  be  shopped  once 
per  quarter  initially  to  make  an  impact,  and  then  later,  reduce  the  number  to  twice  per  year.  For 
the  most  part  the  companies  agreed  that  troubled  properties  should  be  shopped  more  often. 

Costs  of  program.  Since  most  of  the  hotel  companies  used  full-time  employees,  the  cost 
of  their  program  was  unquantifiable.  They  did  not  maintain  average  costs  per  evaluation. 
Marriott  paid  their  contracted  callers  $3.10  per  call.  Marriott  estimated  that  their  contract  callers 
made  on  average  250  calls  per  month. 

There  was  very  little  cost  information  from  restaurant  companies  as  well.  Bob  Evans 
stated  that  the  cost  per  shop  varied  based  on  the  size  of  the  contract  and  which  meal  was  being 
shopped.  Applebee’s  paid  customers  that  completed  their  telephone  survey  a  $3.00  coupon  for 
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their  next  visit.  They  also  stated  that  their  CSI  program  cost  about  the  same  as  a  mystery 
shopping  program.  Shoney’s  paid  $7.50  per  shop  before  3:00  p.m.  and  $10.00  after.  All  shops 
for  Shoney’s  Capt  D’s  restaurants  paid  $7.50. 

The  costs  associated  for  consulting  companies  were  divided  between  what  they  charged 
the  client  for  their  service  and  what  they  compensated  their  shoppers.  The  amount  charged  to  the 
clients  for  restaurant  shops  varied  by  company.  Feedback  Plus  charged  $20-$  100  per  shop. 

BMA  quoted  $60  per  shop  and  then  negotiated  from  there.  A  Closer  Look  charged  between  $40 
and  $75.  The  average  rate  charged  by  these  three  companies  was  $57.50.  There  were  fewer 
responses  on  the  amount  charged  for  hotel  shops.  Richey  International  usually  charged  $2,000 
for  a  two-day  shop  and  $6,000  for  a  five-day  or  longer  shop.  A  Closer  Look  charged  between 
$40-$75  for  each  hotel  shop.  The  amount  charged  per  shop  varied  based  on  the  number  of 
shops,  the  level  of  detail  required,  and  the  t5q)e  of  property  being  shopped.  As  a  rule,  budget  or 
quick  service  companies  were  charged  less  while  upscale  companies  were  charged  more. 

The  compensation  to  the  shopper  was  fairly  consistent.  Most  companies  only  reimbursed 
shoppers  for  the  cost  of  their  meal  or  their  one-night  hotel  stay.  Sometimes  a  consulting 
company  paid  the  shopper  more  depending  on  the  agreement  with  the  client. 

Recognition  program.  Three  of  the  five  hotel  companies  had  some  form  of  recognition 
through  their  evaluation  programs.  Marriott  recognized  properties  corporately  for  scoring  a  90% 
or  better  on  their  shops.  A  90%  or  higher  score  received  a  silver  award  and  a  95%  or  better 
received  a  gold  award.  The  Radisson  combined  their  shopping  results  with  customer  comment 
cards  to  create  a  Quality  Performance  Rating  (QPR).  A  certain  score  on  the  QPR  resulted  in  the 
property  being  recognized  as  a  “President’s  Award  Winning  Hotel.”  The  company  awarded  120 
out  of  360  of  their  properties  with  this  recognition  last  year.  Hyatt  tracked  imsolicited  comments 


about  employees  and  rewarded  them  accordingly.  Overall,  individual  awards  were  usually 
handled  at  the  property  level. 
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Only  two  restaurant  companies  had  a  recognition  program.  Bob  Evans  rewarded  all 
employees  of  a  unit  if  it  scored  100%  on  10  of  the  26  annual  shops.  Old  Country  Buffets  also 
had  a  unit  recognition  program  at  the  corporate  level.  All  of  the  restaurant  companies  left 
employee  recognition  to  the  individual  units  to  handle.  The  recognition  program  area  was  not 
applicable  to  consulting  companies,  although  Feedback  Plus  did  mention  that  they  could  assist 
their  clients  in  setting  up  a  recognition  program  to  coincide  with  the  mystery  shopping  program. 

Problem  areas.  Most  restaurant  and  consulting  companies  identified  getting  the  shop 
completed  on  time  or  at  all  as  the  main  problem.  They  also  mentioned  program  set-up, 
measuring  service  timing,  and  shoppers  understanding  new  questions  on  the  forms.  Hotel 
companies  identified  consistency  between  evaluators  and  properties  not  being  prepared  as  the 
main  problem  areas.  Shoppers  being  discovered  was  not  a  problem  area  identified. 

Other  comments  and  unexpected  findings.  The  responses  for  the  most  part  followed 
closely  with  the  literature  review.  However,  Applebee’s  used  an  interesting  variance  to  mystery 
shopping  called  the  consumer  service  index.  This  program  selected  every  30*  customer  through 
the  POS  system.  Thee  customer’s  receipt  had  a  1-800  number  for  them  to  call  and  a  $3.00 
coupon.  The  coupon  could  not  be  used  unless  the  customer  called  the  number  and  answered  14 
questions  within  72  hours  of  the  meal.  Applebee’s  was  pleased  with  the  results  of  their  CSI 
program.  They  were  able  to  discover  and  fix  a  product  temperature  problem  within  three  days  of 
the  product’s  roll-out.  The  cost  of  the  program  was  similar  to  mystery  shopping,  however,  it  was 
able  to  obtain  a  larger  sample  much  faster.  The  biggest  downfall  of  the  CSI  was  no  recognition 
program  for  employees. 
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Bob  Evans  was  very  pleased  with  their  mystery  shopping  program.  The  company 
conducted  17,000  shops  for  the  400-unit  chain  last  year.  The  key  to  its  success  was  having  clear 


standards.  They  were  not  really  measuring  customer  satisfaction  as  much  as  they  were 
measuring  compliance  with  corporate  standards.  They  believed  that  the  latter  would  have  a 
positive  effect  on  the  former.  They  did  not  use  any  written  comments  on  their  forms. 

Steve  Snedegar,  President  of  Guest  Perception  Inc.,  stated  that  there  were  at  least  400 
other  consulting  companies  that  he  was  competing  with.  This  added  pressure  on  price  and 
service.  He  noted  that  it  was  easy  to  start  a  mystery  shopping  company,  but  it  took  a  lot  of 
capital  to  land  the  big  accounts. 

Wright-Patterson  Air  Force  Base  Mystery  Shopping  Program  Results 

A  total  of  43  shops  were  completed.  Shoppers  were  provided  program  critique  forms 
with  each  shop.  The  shoppers  filled  out  a  total  of  36  critique  forms.  Several  of  the  shoppers, 
who  completed  more  than  one  shop,  completed  only  one  critique  form.  Of  the  36  completed 
critiques,  23  had  written  comments  (See  Appendix  S  for  the  combined  results).  These  comments 
are  summarized  below  with  additional  observations  about  the  effectiveness  and  problems  of  the 
training  program,  shopping  instructions,  evaluation  forms.  Guide  to  Mystery  Shopping,  and  the 
effect  of  the  program  on  the  shoppers’  opinion  of  the  Services  Division.  In  addition,  a  summary 
of  the  effectiveness  and  problems  occurring  with  the  recruiting,  recognition,  database,  costs,  and 
implementation  of  the  mystery  shopping  program  at  Wright-Patterson  AFB  will  follow.  Finally, 
a  summary  of  the  overall  program  will  be  provided  including  a  discussion  of  the  quality  of  data 


obtained. 
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Training  Program 

Thirty-five  of  the  thirty-six  shoppers  (one  non-response)  said  the  training  information 
was  effective.  Six  shoppers  felt  that  the  training  answered  all  of  their  questions  and  was  very 
informative.  One  shopper  felt  that  only  experience  could  truly  prepare  someone  for  shopping. 
One  shopper  wanted  more  information  about  the  88*  Services  Division. 

The  information  provided  in  the  training  class  was  based  mostly  from  the  Services  Guide 
to  Mystery  Shopping.  However,  there  were  a  few  items  that  came  up  during  the  classes  that 
should  have  been  provided  in  the  class.  The  marketing  e-mail  address  could  have  been  given  to 
the  trainees  so  that  they  could  send  a  quick  e-mail  to  the  Services  Marketing  Division.  This 
would  have  helped  with  setting  up  the  e-mail  listing  of  the  shoppers  and  increased 
communication  earlier  on  in  the  program.  This  could  also  help  trainees  that  did  not  know  what 
their  e-mail  address  was.  This  was  the  case  for  thirteen  out  of  thirty-two  shoppers.  The  shoppers 
also  could  have  been  told  to  keep  a  copy  of  their  completed  shopping  form  and  receipt  just  in 
case  there  were  questions  about  either  one.  More  information  could  have  been  provided  on  how 
to  obtain  an  employee’s  name.  This  topic  was  emphasized  more  in  the  later  training  sessions, 
but  still  did  not  achieve  the  desired  results.  Several  of  the  shoppers  did  not  check  the  restrooms 
during  the  shop  because  the  immediate  activity  did  not  have  any.  This  information  could  be 
included  in  the  training  session  to  let  shoppers  know  how  to  handle  these  situations. 

Thirty-five  of  the  thirty-six  shoppers  (one  non-response)  said  the  training  techniques  used 
in  the  class  were  effective.  Six  shoppers  felt  the  class  was  enjoyable  and  very  helpful.  One 
shopper  suggested  that  the  trainers  use  role-playing  to  help  teach  shoppers  how  to  obtain 
employee  names.  Another  shopper  suggested  bringing  in  experienced  shoppers  to  discuss  their 


knowledge  with  the  class.  One  shopper  felt  the  class  would  be  more  effective  if  there  was  a 
better  mix  of  males  and  females.  He  was  the  only  male  in  a  class  of  eleven  people. 
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The  classes  were  taught  using  an  informal  discussion  technique.  This  method  worked 
well  to  make  the  classes  comfortable  and  to  encourage  discussion  and  questions.  Overheads 
could  be  used  to  provide  a  visual  overview  of  the  mystery  shopping  program  and  of  the  class 
itself.  Role-playing  could  work  well  to  show  shoppers  what  to  expect  and  how  to  implement 
strategies  such  as  obtaining  a  name  or  measuring  times  for  drink  and  meal  delivery. 

Written  Shopping  Instructions 

Thirty-six  out  of  thirty-six  shoppers  said  the  written  instructions  provided  with  the 
evaluation  forms  were  useful  and  easy  to  follow.  However,  several  of  the  written  comments 
showed  that  there  were  several  areas  that  needed  improvement.  Six  respondents  were  pleased 
with  the  instractions  as  they  were  written.  Three  respondents  had  difficulty  with  the  lodging 
instructions.  The  shoppers  did  not  have  enough  information  to  easily  obtain  a  reservation  for 
their  shop  without  being  detected  as  a  shopper.  There  was  also  some  confusion  about  how  the 
bill  would  be  handled.  Two  shoppers  found  the  instructions  did  not  match  very  well  with  the 
location  that  was  being  shopped,  making  it  difficult  to  complete  the  shop.  One  shopper  was  only 
provided  48  hours  to  complete  a  Friday  night  shop  because  the  form  due  date  came  before  the 
next  Friday  night. 

Overall  the  written  shopping  instructions  worked  well.  The  instructions  were  intended  to 
be  a  quick  reference  for  the  shop,  however,  there  were  so  many  important  reminders  that  they 
quickly  filled  the  single  page  thus  limiting  the  amount  of  specific  instructions  for  the  activity.  In 
addition,  the  specific  instructions  on  what  the  shopper  needed  to  do  for  each  shop  was  minimal 
due  to  the  limited  feedback  received  from  the  managers  of  the  activities. 
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The  lodging  instructions  were  less  effective  than  the  foodservice  instructions.  This  was 
due  in  part  to  the  complicated  directions,  related  to  AF  travel  regulations,  that  the  shopper 
needed  to  follow  to  make  reservations  and  obtain  a  room.  As  a  result,  the  lodging  instructions 
could  have  been  two  pages,  or  the  generic  shopping  reminders  could  have  been  reduced. 

Evaluation  Forms 

Thirty-six  out  of  thirty-six  respondents  said  the  evaluation  forms  were  effective  for  the 
shops.  Four  of  the  respondents  providing  comments  felt  the  forms  were  great  the  way  they  were. 
There  were  several  suggestions  made  to  improve  the  forms.  Three  respondents  felt  that  the 
forms  for  the  canteens  and  buffet  style  restaurants  did  not  match  very  well.  They  felt  that  these 
forms  were  more  geared  toward  a  table-service  dinner  shop.  A  couple  of  the  respondents  noted 
confusion  with  the  questions  that  asked  two  or  more  items.  For  example,  one  questions  was 
worded  “Did  the  employee  have  a  friendly  demeanor  and  extend  a  greeting.”  The  shopper  felt 
that  an  employee  could  do  one  part  and  not  the  other,  making  it  confusing  whether  to  mark 
“Yes”  or  “No.”  Although  this  situation  was  discussed  in  the  training  session,  it  caused  some 
confusion  with  the  shoppers.  A  couple  of  the  questions  confused  several  shoppers.  For  example, 
at  activities  with  an  a  la  carte  service,  the  shoppers  understood  the  question  “Were  you  quickly 
greeted  upon  your  arrival  and  made  to  feel  welcome”  to  mean  from  the  time  they  entered  the 
facility  instead  of  when  they  approached  the  service  counter. 

Other  suggestions  included  adding  a  question  on  disabled  access/facilities,  numbering  the 
questions  on  the  form  for  easier  referencing  in  the  comment  sections,  and  providing  the  forms  on 
disk  or  the  web  to  make  them  easier  to  complete.  Many  of  the  suggestions  provided  would 
increase  the  length  and  complexity  of  the  forms.  Several  shoppers  thought  that  the  length  of  the 
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pre-call  visit  should  be  recorded,  as  well  as  the  in  and  out  time  at  the  bar  separate  from  the  dining 
times. 

Overall,  the  forms  worked  well.  Several  shoppers  had  difficulty  placing  the  ratings  in  the 
correct  locations,  despite  having  samples  to  follow.  The  design  of  the  form  may  have  had  an 
effect  on  the  number  of  unanswered  questions.  In  several  cases  the  shopper  skipped  an  entire 
section.  On  several  of  the  questions  that  required  both  a  Yes/No  answer  and  a  rating,  one  or  the 
other  (usually  the  rating)  was  left  blank.  The  comments  often  helped  clarify  hard  to  read  or 
confusing  ratings.  The  forms  generated  a  lot  of  insightful  comments,  which  was  much  better 
than  expected. 

The  Services  Guide  to  Mystery  Shopping 

The  Guide  was  well  received.  Thirty-six  out  of  thirty-six  respondents  said  the  Guide  was 
a  useful  reference.  Six  shoppers  said  the  guide  was  well  organized  and  useful.  One  shopper 
requested  that  a  binder  or  folder  be  provided  to  keep  the  Guide  and  other  materials  together. 
Another  suggested  that  the  Services  Web  site  URL  be  included  in  the  guide  for  reference. 

The  Guide  was  designed  to  correspond  with  the  training  course  so  that  shoppers  could 
reference  any  area  after  they  left  the  training.  As  a  result,  all  additional  information  added  to  the 
training  class  could  also  be  included  in  the  Guide  and  vice  versa.  The  Guide  should  include  e- 
mail  and  web  site  information,  a  recommendation  to  keep  copies  of  completed  form  and  receipts, 
a  reminder  to  check  restrooms  even  if  the  immediate  facility  does  not  have  one,  and  a  reminder 
not  to  rate  anything  that  the  shopper  did  not  actually  see.  The  completed  examples  in  the  back  of 
the  Guide  were  very  useful  in  explaining  how  to  properly  complete  the  forms. 
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Shopper  Opinion  of  the  Services  Division 

Thirty  out  of  thirty-six  of  the  survey  respondents  (six  non-responses)  said  that  the 
mystery  shopping  program  positively  affected  their  opinion  of  the  88*  Services  Division.  Five 
shoppers  noted  that  the  program  showed  that  the  Division  cared  about  what  the  customers 
thought  and  showed  that  they  were  trying  to  improve  service.  Two  respondents  also  said  that  it 
was  good  that  the  Division  was  using  the  program  to  recognize  employees. 

Recruiting 

The  recruiting  program  worked  very  well.  Within  one  week  of  initial  advertisements, 
enough  shoppers  were  recruited  for  the  first  full  class.  A  steady  stream  of  interested  people 
contacted  the  Marketing  Dept,  to  sign  up  for  the  program.  After,  thirty  shoppers  were  trained, 
there  were  still  numerous  names  on  the  waiting  list  to  become  shoppers.  As  a  result,  the 
recruiting  efforts  were  reduced  to  prevent  a  bigger  backlog.  Shoppers  on  the  waiting  list  were 
notified  that  they  would  not  be  trained  until  later  in  the  year.  The  electronic  Weekly  Bulletin 
was  the  main  source  for  attracting  shoppers.  The  recruiting  brochures  were  also  effective  in 
attracting  several  shoppers. 

One  area  of  concern  was  the  demographic  characteristics  of  the  shoppers  as  a  group.  No 
quotas  were  set  when  the  program  was  designed,  but  it  quickly  became  apparent  that  several 
groups  were  underrepresented.  Only  22%  of  the  shoppers  were  male.  This  was  not 
representative  of  the  base  population,  which  was  predominately  male.  Even  more  of  a  concern 
was  the  lack  of  military  shoppers.  There  were  only  two  active  duty  officers,  two  active  duty 
enlisted,  three  retirees,  and  one  reservist.  The  majority  of  the  shoppers  (59%)  were  government 
civilians.  Although  the  civilians  are  the  largest  demographic  group  on  the  base,  they  are  not  the 
primary  customers  in  several  activities  such  as  lodging,  the  nnilitary  dining  facility,  the  Officer’s 
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Club,  the  bowling  center  and  one  of  the  golf  courses.  As  a  result,  it  was  difficult  to  find  the  right 
shoppers  to  complete  several  of  the  shops.  The  lack  of  military  shoppers  that  were  willing  to 
shop  lodging  was  the  main  reason  for  completing  only  seven  of  the  ten  scheduled  shops.  One- 
on-one  training  sessions  were  conducted  to  increase  the  number  of  military  shoppers. 

Recognition 

The  recognition  program  was  more  difficult  to  manage  than  expected.  Even  with  a  first 
name  and  a  good  description,  it  was  a  lot  of  work  to  identify  the  full  name  of  the  employee  in 
order  to  create  the  recognition  certificate.  As  a  result,  the  employees  who  were  not  rated  as  a  “5” 
were  not  researched  to  discover  their  full  names.  It  was  assumed  that  the  activity  managers 
would  know  who  these  employees  were.  There  were  53  employees  that  were  rated  as  a  “5”  on 
the  evaluation  forms.  When  one  employee  was  named  more  than  once  on  a  single  evaluation 
form,  he  or  she  only  received  one  certificate.  Five  of  the  employees  recognized  as  “5”s  were  not 
awarded  a  certificate  because  the  shopper  did  not  obtain  a  name  or  a  good  enough  description. 
The  awards  were  slow  to  be  presented  to  the  employees  because  of  the  delay  in  finding  the 
names,  creating  the  personalized  certificates,  and  obtaining  the  Services  Director’s  signature  on 
the  certificate  and  time-off  award.  No  feedback  was  obtained  from  the  managers  or  employees 
on  the  effectiveness  of  the  recognition  program. 

Costs  of  Program 

The  exact  cost  of  the  entire  program  could  not  be  accurately  determined  because  several 
of  the  costs  were  not  tracked.  These  included  personnel  and  marketing  material  costs.  The 
actual  cost  of  the  shop  reimbursements  was  $290.27  for  the  43  shops.  The  budgeted 
reimbursement  amount  for  these  shops  was  $423.00.  The  actual  amount  was  lower  because 
many  of  the  shoppers  did  not  use  the  maximum  amount  for  each  shop.  In  addition,  six  shoppers 
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did  not  tum-in  a  receipt  and  were  not  provided  a  reimbursement.  The  lodging  actual 
reimbursement  amount  was  higher  than  the  budgeted  amount.  This  was  caused  mostly  by  a 
sharp  increase  in  room  rates  effective  1  October  1998.  In  addition,  two  shoppers  were 
unintentionally  reimbursed  higher  amounts  than  authorized  because  their  receipt  reflected  the 
cost  of  two  people  instead  on  one.  The  actual  gift  certificate  cost  of  $390  closely  matched  the 
budgeted  amount  of  $430.  In  several  cases  the  shopper  did  not  want  their  gift  certificates.  The 
$390  also  did  not  reflect  the  amount  truly  spent  because  many  of  the  certificates  were  used  for 
less  than  their  face  value  or  not  at  all.  This  dollar  total  was  not  available  because  the  expiration 
date  on  the  certificates  had  not  expired. 

The  total  direct  cost  of  the  43  shops  was  $680.27.  The  average  direct  cost  per  shop  came 
to  $15.82.  The  recognition  expenses  of  the  program  included  lapel  pins  ($1.30  per  pin  for  a  total 
of  $62.40),  the  time-off  awards  (48  employees  at  and  average  of  $8  per  hour  for  a  total  of  $384), 
and  certificates  ($1  each  for  a  total  of  $48).  The  total  recognition  costs  for  the  43  shops  was 
$494.40.  The  average  cost  per  shop,  not  including  overhead,  was  approximately  $27.32  per 
shop.  The  overhead  costs,  especially  the  personnel  expenses,  could  be  reduced  as  the  program 
progressed  because  the  database  will  handle  many  of  the  functions  currently  performed  by  hand, 
and  the  program  manager  will  become  more  adept  at  administering  the  program. 

Database 

The  “Shoppy”  Excel  database  program  was  very  useful  in  tying  all  aspects  of  the 
program  together.  The  program  controlled  the  entry  of  data  into  the  program  to  help  reduce 
entry  errors  and  faulty  data.  The  Visual  Basic  controlled  functions  saved  a  lot  of  time  and  errors 
by  automatically  sorting  all  of  the  similar  data  into  several  spreadsheets.  The  program  also  saved 
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time  by  automatically  clearing  the  forms  for  the  next  entry  or  for  printing.  The  program  did  not 
reach  its  full  potential  by  the  completion  of  this  project. 

A  downfall  of  the  program  was  its  complexity.  If  a  problem  arose  with  the  program,  it 
would  be  difficult  for  a  non-programmer  to  identify  and  fix.  If  another  base  used  this  program, 
they  would  save  a  lot  more  time  in  setting  up  and  running  their  program  than  Wright-Patterson 
AFBdid. 

Program  Implementation 

Administering  the  program  was  the  most  difficult  part.  The  program  manager  did  not 
have  the  detailed  knowledge  of  the  program  as  the  program  designer  did.  Since  the  program 
designer  worked  three  hours  away,  it  was  difficult  to  coordinate  program  changes.  As  a  result, 
there  were  several  delays  in  the  training  of  shoppers,  assigning  shops,  and  implementing  the 
database  program.  There  were  several  changes  in  the  procedures  of  the  program  that  also  slowed 
the  project.  One  was  the  change  from  gift  certificates  only  to  reimbursement  and  gift 
certificates.  Another  was  an  increase  in  the  amount  of  functions  the  database  program  would 
accomplish. 

The  large  number  of  shops  during  the  short  timeframe  also  caused  some  difficulties.  The 
procedures  were  barely  in  place  when  the  first  wave  of  shops  was  being  turned  in.  This 
overwhelmed  the  system  preventing  the  completed  forms  from  being  reviewed  as  closely  as  they 
should  have  been.  This  also  made  it  difficult  to  adjust  procedures  for  the  future  shops.  By  the 
time  the  first  set  of  shops  were  being  scrutinized  the  second  set  had  already  been  assigned.  In 
addition,  the  shopper  e-mail  list  had  not  been  set  up  yet  making  quick  communication  with  the 
shoppers  more  difficult.  If  the  program  was  working  on  a  normal  schedule  then  there  would 
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have  been  fewer  shops  to  handle  each  week  and  the  program  manager  would  have  been  more 
familiar  with  the  procedures  making  the  completed  shops  easier  to  process. 

Program  Flexibility 

The  entire  program  was  written  as  genetically  as  possible  to  meet  the  needs  of  the 
Wright-Patterson  AFB  Services  Division  and  the  Services  functions  at  other  bases  with  little 
modification.  All  of  the  forms  were  given  generic  titles.  Only  minor  modifications  would  be 
required  to  the  location  field  of  the  forms  to  make  them  fit  another  base.  The  front  cover  and 
location  specific  information  would  need  to  be  changed  in  the  Services  Guide  to  Mystery 
Shopping.  Each  set  of  shopping  instructions  would  have  to  be  reviewed  to  make  sure  they  fit  the 
new  base’s  facilities.  The  recruiting  materials  would  need  to  be  reprinted,  but  the  main 
information  and  logos  could  remain  the  same. 

There  would  be  almost  no  Visual  Basic  program  changes  required  for  another  base  to 
utilize  the  database  program  unless  the  new  base  did  not  want  to  measure  the  same  twenty-one 
items  on  the  results  database.  Changing  one  or  two  questions  could  work,  but  it  would  require 
someone  who  fully  understood  the  Visual  Basic  programming  language.  Any  more  significant 
changes  would  require  a  full  rewrite  of  the  program  to  be  effective.  The  more  functions  the 
program  was  able  to  do,  the  less  flexible  it  would  be  to  modify. 

Comparison  with  Commercial  Programs 

This  Air  Force  mystery  shopping  program  varied  a  lot  from  commercial  programs.  Very 
few  commercial  companies  ran  an  internal  program,  but  hired  non-company  employees  to 
complete  the  shops.  Of  the  commercial  programs  that  used  non-company  shoppers,  the 
recruiting,  training,  compensation,  and  recognition  areas  were  much  different.  The  AF  program 
recruited  with  several  media  forms,  but  did  not  use  referrals.  This  program  also  fully  trained  all 
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shoppers  before  their  first  shop  in  a  classroom  environment,  while  most  commercial  companies 
either  expected  the  shoppers  to  have  experience  or  trained  them  with  written  instructions  before 
their  shop  and  then  with  feedback  afterwards.  The  AF  program  compensated  all  shops  with 
$10.00  gift  certificates  above  the  reimbursement  amount  for  the  shop.  Most  commercial 
companies  only  compensated  the  shoppers  with  a  free  meal  or  overnight  stay.  The  AF 
recognition  program  focused  on  rewarding  individuals  for  outstanding  performance,  while  most 
commercial  programs  rewarded  at  the  unit  level. 

There  were  several  areas  of  the  AF  program  that  was  similar  with  commercial  programs. 
These  areas  included  evaluation  forms,  shopping  instructions  and  feedback  to  shoppers.  Similar 
to  commercial  companies  the  evaluation  forms  were  comprised  mostly  of  objective  Yes/No 
questions  with  a  few  rated  subjective  questions.  The  shopping  instructions  followed  a  similar 
format  to  commercial  instructions  by  reminding  the  shopper  about  important  areas  to  be 
shopped.  Like  most  commercial  programs,  the  AF  program  provided  shoppers  with  feedback  on 
their  shops  to  help  them  achieve  better  results  on  future  shops. 

Overall  Program  Summary 

The  data  obtained  by  the  program  was  appeared  to  be  good  quality  data.  For  the  most 
part,  the  written  comments  validated  the  objective  questions  and  ratings.  The  shoppers,  as 
indicated  by  the  mystery  shopping  results  database  (see  Appendix  P),  completed  most  of  the 
questions.  There  were  a  total  of  29  blank  cells  (3.2%)  in  the  entire  database.  The  field  with  the 
most  blanks  was  “Public  Restroom  Cleanliness”  with  nine.  The  field  with  the  second  most 
blanks  was  “Employee  Demeanor  and  Greeting”  with  seven.  Thirteen  fields  did  not  have  any 
blank  entries.  If  the  information  from  the  recognition  database  is  included  (see  Appendix  T)  then 


the  29  blank  cells  increases  to  59  (4.7%).  The  recognition  database  blanks  were  attributed 
mostly  to  shoppers  not  obtaining  employees  names. 
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Most  of  the  ratings  were  3, 4  or  5.  The  rated  fields  ranged  between  3.91  and  4.31,  with 
an  average  score  of  4.04.  The  percentage  of  “Yes”  responses  ranged  from  45%  for  “Uniform 
and  Nametag?”  to  100%  for  “Identify  Activity  Name?” 

Of  the  original  50  scheduled  shops  identified  to  test  the  program,  46  of  them  were 
actually  assigned  to  shoppers.  Of  those,  only  three  shops  were  not  completed  on  time.  Two  of 
these  were  missed  due  to  a  medical  emergency  that  afflicted  a  shopper.  When  the  shopper 
returned  from  the  hospital  she  let  the  program  manager  know  that  she  could  not  complete  the 
shops.  So,  in  essence  only  one  assigned  shop  was  not  completed  do  to  shopper  error.  Therefore, 
the  shop  miss  rate  was  only  2.3%. 

Overall  the  program  was  successful.  The  Services  Division  received  detailed  feedback 
on  their  foodservice  and  lodging  operations.  The  managers  received  specific  written  comments, 
ratings,  and  employee  recognition  for  their  activities.  The  Services  Division  received 
consolidated  data  on  how  all  of  its  activities  were  serving  the  community  as  a  whole  in  the  key 
areas  defined  by  the  customers.  In  addition,  the  program  had  been  created  generically  enough  to 
allow  other  bases  to  modify  it  to  meet  their  needs.  All  of  this  was  accomplished  under  budget. 
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CHAPTER  V 

DISCUSSION,  CONCLUSIONS  AND  RECOMMENDATIONS 
Telephone  Survey  Discussion  and  Comparison  with  Literature  Review 

The  telephone  survey  corresponded  closely  with  the  literature  review.  One  area  that 
differed  was  the  frequency  of  restaurant  shops  from  one  per  month  in  the  literature  review  to  four 
per  month  in  the  survey.  The  frequency  of  hotel  shops  was  once  per  month  in  the  literature 
compared  to  only  twice  per  year  in  the  survey. 

The  telephone  survey  had  several  limitations.  First,  the  sample  was  not  randomly 
selected.  Second,  the  participants  were  hesitant  to  provide  detailed  company  information  or 
strategies.  In  general,  the  same  survey  was  verbally  presented  over  the  telephone  to  each  subject. 
However,  many  of  the  responses  led  to  unplanned  follow-up  questions  that  did  not  occur  in  all 
interviews. 

The  telephone  survey  identified  late  or  missed  shops  as  the  primary  problem  encountered 
by  external  mystery  shopping  programs.  One  company  listed  their  miss  rate  as  10%  of  their  total 
shops.  The  miss  rate  for  the  Air  Force  program  was  only  2.3%  in  comparison.  One  possible 
explanation  could  be  the  Air  Force’s  more  detailed  training  program  that  occurred  before  the 
shops  took  place.  Although  this  shop  miss  rate  figure  shows  a  positive  comparison,  the  Air 
Force  program  may  not  have  been  operating  long  enough  to  obtain  the  true  average  shop  miss 
rate. 

Use  of  the  Mystery  Shopper  Program  Critique 

The  critique  was  not  designed  to  be  statistically  sound.  Instead  it  served  to  collect 
comments  and  suggestions  about  how  to  improve  the  mystery  shopping  program.  The  comments 
tended  to  lean  toward  the  positive  side.  This  was  similar  to  the  average  ratings  that  the  shoppers 
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gave  the  different  activities.  Every  shopper  was  presented  with  the  purpose  of  the  critique  and 
was  required  to  return  the  completed  critique  with  the  completed  evaluation  form  for  each  shop. 
Although  36  completed  critiques  were  returned,  only  23  had  conunents.  Many  of  these 
comments  were  superficial  and  only  validated  the  responses  to  the  critique’s  Yes/No  questions. 
The  critique  would  have  been  more  effective  if  the  program  manager  had  reviewed  the  critique 
and  asked  the  shopper  to  elaborate  on  the  questions  or  obtain  clarification  of  the  comments 
provided. 

Interviews  would  have  been  a  good  approach  to  obtain  more  information  from  the 
shoppers  about  each  element  of  the  program.  Follow-up  questions  could  have  been  asked  which 
might  have  revealed  feedback  on  other  areas  of  the  program  not  covered  by  the  critique.  The 
results  from  the  critique  should  be  viewed  with  caution.  Although,  the  shoppers  may  have 
numerous  good  suggestions,  they  do  not  understand  all  of  the  requirements  of  the  program  and 
the  limitations  created  by  funding  shortages  and  AF  regulations.  The  shoppers’  inputs  should  be 
balanced  with  the  program  manager’s  ideas,  and  feedback  from  the  employees  and  managers. 
Training  Program  Discussion 

The  training  program  was  effective  in  preparing  the  shoppers  for  their  duties.  The  low 
number  of  blank  fields  on  the  database  and  the  high  number  of  shoppers  who  identified  the 
training  program  as  effective  were  indicators  that  the  training  worked  well. 

The  training  program  differed  from  the  commercial  restaurant  mystery  shopping 
programs  by  training  the  shoppers  before  they  performed  their  first  shop  instead  of  after.  This 
pre-training  approach  was  better  than  the  post-training  approach  because  the  shoppers 
understood  what  they  needed  to  know  before  they  completed  the  shop.  The  post-training 
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approach  identified  errors  in  the  shopping  evaluation  after-the-fact.  This  would  indicate  that  the 
first  couple  of  shops  were  less  reliable  with  the  post-training  approach. 

There  were  several  areas  that  should  be  improved  with  the  training  program.  These  were 
instructional  techniques,  emphasis  on  ratings,  and  more  information.  Role-playing  should  be 
included  in  the  instruction.  This  instructional  technique  would  better  simulate  the  actual 
environment  of  the  shopping  experience  over  discussion  of  the  topic.  This  change  should  help 
reduce  the  number  of  blank  employee  name  fields  on  the  completed  evaluation  forms. 

The  second  change  in  training  should  be  a  reduced  emphasis  on  the  rating  scale.  In  an 
effort  to  encourage  recognition  of  employees,  the  training  put  too  much  emphasis  on  giving  high 
scores  (especially  “5”s  to  the  employees).  The  shoppers  should  understand  the  importance  of 
their  ratings  on  the  recognition  program,  but  also  be  honest  about  the  actual  performance  of  the 
activity.  One  of  the  ways  that  this  could  be  better  trained  is  by  using  examples  of  employee 
behavior  and  what  the  appropriate  rating  should  be.  For  example,  an  employee  might  be  very 
friendly  and  competent,  but  if  the  employee  isn’t  wearing  a  nametag  (which  is  mandatory  and 
easy  to  do),  then  they  probably  shouldn’t  be  considered  for  a  “5”  rating.  The  shoppers  need  to 
understand  that  the  questions  on  the  form  are  based  on  what  Services  wants  the  employees  to  do. 
If  those  areas  aren’t  met,  even  though  the  shopper  thinks  the  employees  are  nice,  they  shouldn’t 
rate  them  as  5s. 

Finally,  the  training  should  include  several  areas  that  were  missing  in  the  Services  Guide 
to  Mystery  Shopping  and  subsequently  in  the  training  classes.  The  training  should  cover  how  to 
shop  lodging  in  more  detail.  The  training  briefly  identified  the  unique  lodging  situation,  but  did 
not  go  into  detail  on  how  to  obtain  a  reservation  and  rate  the  housekeeper.  These  areas  were 
more  challenging  and  needed  to  be  clearly  explained.  The  training  class  should  also  explain  how 
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a  shopper  should  handle  problems  during  the  shop.  They  should  be  provided  suggestions  on  how 
to  handle  unexpected  situations,  such  as  a  buffet  at  a  table  service  activity.  This  will  increase  the 
shopper’s  confidence  and  let  them  know  that  unexpected  things  could  occur.  The  shoppers 
should  also  be  informed  to  keep  a  copy  of  their  receipt  and  completed  shopping  form.  On  at 
least  four  occasions,  the  program  manager  called  the  shopper  to  obtain  clarification  on  a  form.  It 
was  difficult  to  obtain  detailed  information  because  the  shoppers  did  not  have  a  copy  of  the  shop 
to  reference. 

The  standardized  training  had  both  positive  and  negative  effects.  On  the  positive  side, 
every  shopper  was  armed  with  the  same  information,  increasing  the  reliability  of  the  data 
collected.  The  standardized  training  allowed  the  shoppers  to  hear  specific  examples  of  what 
worked  and  what  to  avoid.  On  the  negative  side,  the  standardized  training  caused  the  shoppers  to 
limit  their  responses.  This  was  especially  evident  with  the  overall  high  ratings.  The  shoppers 
were  encouraged  to  give  “5”  ratings.  As  a  result,  when  there  were  things  wrong  with  the  facility, 
the  shoppers  gave  4s  and  3s  instead  of  2s  and  Is  in  most  cases.  The  description  of  the  scale 
should  be  changed  to  encourage  the  shoppers  to  use  the  entire  scale.  The  description  of  “5” 
ratings  should  be  changed  from  “Outstanding  (The  way  it  should  be.  Please  use  them!  Don’t 
save  5’s  for  perfection.)”  to  “Outstanding  (World  class  service  and  quality.  The  way  it  should 
be).”  This  change  will  help  reduce  the  trainer’s  influence  over  the  ratings  that  are  given.  In 
addition,  the  lower  end  of  the  scale  should  be  used  to  describe  areas  needing  attention.  Including 
Is  in  the  example  forms  discussed  in  class  will  better  relay  this  point. 

Written  Shopping  Instructions  Discussion 

The  written  shopping  instructions  were  generally  effective.  However  there  were  several 
situations  where  the  instractions  did  not  adequately  prepare  the  shopper.  The  worst  situation  was 
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lodging.  There  were  several  problems  encountered  by  the  shoppers  in  obtaining  a  reservation. 
The  instructions  assumed  the  shoppers  were  familiar  with  the  lodging  reservation  process  and 
could  easily  pretend  to  be  an  actual  customer  on  military  orders.  Instead,  the  shoppers  were 
frustrated  and  the  quality  of  the  data  was  jeopardized.  Several  of  the  shoppers  performing  the 
lodging  shops  informed  the  program  manager  that  they  almost  did  not  complete  the  shop  because 
of  the  difficulty  they  had  making  reservations.  In  fact,  two  of  the  shoppers  had  to  call  back  a 
second  time  to  complete  the  reservation  process.  By  not  knowing  what  to  say  to  the  reservation 
clerks,  the  shoppers  jeopardized  the  success  of  the  shop  and  increased  the  chance  of  being 
discovered. 

The  lodging  instructions  were  limited  to  fit  on  a  single  page.  As  a  result,  detailed 
instructions  were  limited.  The  instructions  should  be  lengthened  to  include  better  instructions, 
and  the  shoppers  should  be  provided  with  the  questions  that  they  will  be  asked  by  the 
reservationist. 

Four  shoppers  had  difficulty  completing  the  pre-visit  call  using  the  instructions.  Two 
telephone  numbers  were  incorrect.  Another  location  was  part  of  an  electronic  answering  system. 
This  system  did  not  have  the  snack  bar  as  one  of  the  menu  options.  If  the  caller  did  not  already 
know  the  right  extension  then  they  would  not  be  able  to  complete  the  call.  There  were  five 
canteen  snack  bars  that  only  have  one  employee  working.  When  the  employee  was  busy  cooking 
and  serving  the  customers,  he/she  was  unable  to  answer  the  telephone.  As  a  result  several  calls 
were  either  unanswered  or  answered  by  an  answering  machine.  The  Services  Division  should 
look  at  this  problem  and  decide  if  it  wants  these  phones  answered  within  the  three-ring  policy  or 


if  the  instructions  need  to  be  modified  for  these  activities. 
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Evaluation  Forms  Discussion 

There  were  several  questions  on  the  evaluation  forms  that  rated  more  than  one  specific 
item.  This  was  identified  as  a  problem  by  several  shoppers  because  they  often  found  that  one 
part  of  the  question  was  being  done  correctly  while  the  other  part  was  not  being  done.  This 
confused  the  shoppers  when  completing  the  forms.  Several  of  these  questions  were  marked  both 
“Yes”  and  “No”  or  not  rated  at  all.  The  shoppers  were  trained  to  mark  the  question  as  “No”  if 
one  part  of  the  question  was  not  being  done  and  then  explain  the  situation  in  the  comments.  This 
allowed  more  areas  to  be  rated  on  the  form  while  keeping  them  within  two  pages.  There  was 
little  difficulty  understanding  the  information  if  the  original  forms  were  available  for  review. 
However,  problems  arose  on  the  dual  questions  that  were  transferred  to  the  results  database.  The 
comments  were  no  longer  available  to  explain  which  part  of  the  question  was  missed.  For 
example,  the  “Uniform  and  Nametag?”  field  on  the  results  database  indicated  that  55%  of  the 
employees  were  not  in  compliance.  From  this  information,  it  was  impossible  to  determine  if  the 
problem  was  with  the  nametags  or  the  uniforms.  After  a  review  of  the  forms,  99%  of  the 
problems  were  with  the  nametags  not  being  worn.  With  this  new  information,  the  problem  could 
be  better  addressed.  Each  of  the  multiple-area  questions  should  be  reviewed  to  find  out  if  there 
was  a  particular  trend  effecting  the  ratings.  These  questions  (especially  if  they  were  transferred 
to  the  results  database)  should  be  reworded  to  better  reflect  a  particular  area  that  was  rated. 

The  last  question  on  the  form  “Based  on  this  experience  would  you  return?”  had  validity 
problems.  Three  of  the  shoppers  checked  this  question  as  “No”  and  then  explained  that  the  only 
reason  they  would  not  return  was  the  distance  to  the  activity.  Since  many  of  the  foodservice 
activities  are  located  in  remote  locations  of  the  base  to  serve  employees  who  work  in  those  areas, 
it  was  not  accurate  to  rate  the  experience  on  travel  time.  The  shoppers  were  not  trained  to  take 
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into  consideration  this  factor,  and  approach  the  facility  from  the  perspective  of  a  regular 
customer.  Instead  they  approached  the  activity  using  their  own  frame  of  reference  for  travel 
distance.  This  problem  should  be  addressed  in  the  training  classes  and  the  Services  Guide  to 
Mystery  Shopping. 

Guide  to  Mystery  Shopping  Discussion 

Although  the  shoppers  found  the  Guide  to  be  very  effective  and  informative,  the  trainers 
identified  several  areas  that  should  be  included  in  the  Guide  to  make  sure  the  shoppers  know 
about  them.  A  more  detailed  explanation  on  how  to  shop  lodging  should  be  added.  This  section 
should  clarify  how  to  make  a  reservation  as  a  shopper  without  having  official  travel  orders.  The 
reservationist  was  required  to  ask  certain  questions  of  the  guest  such  as  what  base  and  unit  the 
guest  is  coming  from,  what  organization  they  will  be  visiting,  and  how  long  they  will  be  staying. 
If  the  shopper  was  prepared  for  these  questions  then  they  can  make  the  reservation  without 
sounding  suspicious  and  possibly  being  discovered.  The  Guide  should  also  explain  how  the 
shopper  should  rate  the  housekeeper  and  obtain  the  housekeeper’s  name.  It  is  likely  that  the 
shopper  may  not  see  the  housekeeper  during  his/her  short  visit.  As  a  result,  the  shopper  needed 
to  call  the  Executive  Housekeeper  afterwards  and  inquire  about  whom  was  cleaning  the  room 
that  day.  This  would  be  a  flag  to  the  Executive  Housekeeper  who  the  shopper  was,  but  since  the 
shopper’s  room  number  is  already  being  given  to  lodging,  this  information  was  no  longer  a 
secret.  To  help  reduce  the  possibility  of  a  shopper  being  recognized,  each  shopper  should  only 
shop  lodging  once  per  year. 

Another  area  that  should  have  been  included  in  the  Guide,  was  how  the  shoppers  should 
handle  unexpected  situations.  The  foodservice  activities  often  hold  special  events  such  as 
buffets,  Mongolian  B-B-Qs,  and  cook-your-own-steak  nights.  These  special  occasions  were 
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difficult  to  predict  and  did  not  fit  the  evaluation  forms  as  well.  The  shopper  should  be  warned  of 
these  events  and  encouraged  to  continue  the  shop  and  adapt  as  much  of  the  form  as  possible  to 
the  event. 

Other  areas  that  should  have  been  addressed  in  the  Guide  were  the  e-mail  address  for  the 
program  manager,  to  always  check  the  restrooms  even  if  the  immediate  activity  didn’t  have  any, 
and  to  only  rate  the  areas  that  the  shopper  actually  saw.  The  restrooms  were  an  area  that  several 
shoppers  did  not  rate.  One  reason  given  was  that  the  immediate  activity  being  shopped  did  not 
have  restrooms.  Since  all  Services  facilities  are  required  to  provide  restroom  facilities  to  the 
customers,  the  shopper  should  know  to  check  the  closest  set  of  restrooms,  even  if  they  are  not 
controlled  by  the  activity  being  shopped. 

Recruiting  Discussion 

The  unbalanced  demographic  mix  of  shoppers  was  a  problem.  There  were  too  many 
government  civilian  shoppers  and  not  enough  military  shoppers.  There  were  even  fewer  active- 
duty  military  shoppers.  Although  most  of  the  base  population  was  comprised  of  government 
civilians,  several  of  the  activities  being  shopped  focused  on  military  customers.  For  example, 
lodging,  the  Officer’s  Club,  and  the  military  golf  course  targeted  military  customers. 

Government  civilians  were  authorized  to  use  these  facilities  but  were  not  the  primary  customers. 
As  a  result,  it  was  difficult  to  locate  enough  military  shoppers  to  shop  these  military  focused 
facilities.  Lodging  was  the  most  difficult.  Three  scheduled  shops  were  not  completed  due  to 
lack  of  military  shoppers. 

The  solutions  to  this  problem  are  a  focused  recruiting  campaign  on  military  members  and 
a  quota  system  for  all  employment  categories.  The  recruiting  campaign  should  include  posters 
and  information  in  the  military  activities  as  well  as  throughout  the  dormitories  and  base  housing 
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areas.  If  a  quota  system  were  used,  then  each  category  of  shopper  would  be  recruited  until 
enough  of  each  type  had  joined.  The  current  system  of  taking  the  first  people  to  sign  up  was 
ineffective  and  created  an  overload  of  civilian  shoppers. 

Another  demographic  challenge  was  the  large  percentage  of  female  shoppers  vs.  male 
shoppers.  Each  demographic  group  has  certain  biases.  To  minimize  the  effect  of  these  biases, 
an  accurate  representation  of  the  percentage  of  males  and  females  on  the  base  should  be  reflected 
in  the  shopper  pool.  As  a  result,  there  should  be  a  larger  percentage  of  male  shoppers.  This  is  a 
difficult  area  to  solve.  The  best  way  to  handle  it  is  to  be  up  front  in  the  recruiting  campaign  by 
announcing  that  shoppers  will  be  selected  to  fill  quotas  that  are  representative  of  the  base 
population.  This  quota  picture  of  the  base  population  could  be  easily  obtained  from  the  base 
personnel  office. 

Recognition  Program  Discussion 

The  success  of  the  recognition  program  was  difficult  to  determine  because  employees 
could  not  be  interviewed  in  time  for  this  project.  The  time-off  award,  pin  and  certificate  were 
unique  recognition  items  that  differed  greatly  from  the  traditional  recognition  used  by  the 
Division.  The  mystery  shopper  recognition  program  significantly  increased  the  number  of 
employees  that  were  being  recognized  within  the  Division.  There  were  47  awards  presented  to 
employees  during  the  two-month  test  period.  This  contributed  to  the  Division’s  goal  of 
increasing  employee  recognition. 

The  AF  program  only  recognized  individual  employees  that  performed  well  during  the 
shop.  Many  of  the  companies  interviewed  in  the  telephone  survey  offered  recognition  programs 
that  rewarded  the  entire  unit  for  achieving  a  certain  level  of  results.  Since  one  of  the  main  goals 
of  the  mystery  shopping  program  and  the  Services  Division  was  to  recognize  employees,  adding 
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a  unit  level  award  would  increase  the  number  of  employees  who  were  eligible  for  recognition 
and  provide  a  positive  incentive  for  managers  to  encourage  their  employees  to  succeed.  In  order 
to  implement  such  a  program,  a  fair  rating  system  would  have  to  be  devised  and  the  activities 
would  need  to  be  shopped  an  equal  number  of  times.  This  would  increase  the  current  mystery 
shopping  budget. 

Comparison  of  Program  Costs  with  Consulting  Companies’  Average  Charges 

From  the  telephone  survey  results,  the  average  price  consulting  companies  charged 
clients  per  shop  was  $57.50.  Multiplied  by  43,  this  fee  would  have  cost  the  Air  Force  $2,473  to 
purchase  an  externally  run  program.  Subtracting  the  known  costs  of  the  Air  Force  program, 
which  was  $1174,  would  leave  $1298  to  cover  the  personnel  and  marketing  materials  cost.  The 
printing  cost  of  the  recruiting  brochures,  advertising  posters,  and  other  printing  costs  did  not 
exceed  $300.  These  costs  were  mostly  one-time  costs  and  would  drop  significantly  for  future 
shops  because  the  pool  of  trained  shoppers  would  already  be  in  place.  It  would  be  safe  to 
assume  that  a  maximum  of  $150  for  marketing  materials  was  attributable  to  the  43  completed 
shops.  Assuming  the  program  manager  was  paid  $10.00  per  hour,  he  or  she  would  be  able  to 
spend  111  hours  administering  the  program  for  these  43  shops.  Assuming  the  program  was 
already  in  place,  the  program  manager  was  trained,  and  a  pool  of  trained  shoppers  was  available, 
it  would  only  require  48  hours  for  the  program  manager  to  administer  the  program  for  43  shops 
over  an  eight- week  period.  Based  on  these  calculations  and  assumptions  the  Air  Force  would 
save  over  $650.  The  savings  depend  directly  on  the  hourly  wage  of  the  program  manager.  Even 
at  $20  per  hour,  the  internal  Air  Force  program  would  be  more  cost  effective. 
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Database  Discussion 

The  database  was  very  successful  in  tracking  the  data,  saving  time,  and  consolidating  the 
entire  program.  The  twenty-one  common  rated  fields  were  harnessed  into  one  database  for  easy 
analysis  and  manipulation.  The  recognition  database  was  critical  in  identifying  the  correct 
employees  for  the  recognition  program.  The  macro-driven  functions  reduced  the  amount  of  time 
and  number  of  errors  on  the  results  database,  the  recognition  database,  and  on  the  budget 
worksheet.  However,  the  database  did  not  reach  its  full  potential.  There  were  many  functions 
that  if  created,  would  have  increased  the  usefulness  of  the  program  and  saved  even  more  time. 
These  functions  included  calculating  averages  of  each  field  and  graphing  data  by  activity,  dates, 
or  both.  The  program  should  also  tie  the  shopping  schedule  with  the  evaluation  forms.  All  the 
user  would  need  to  do  was  select  a  shop,  then  select  a  shopper  (from  a  dropdown  list)  and  the 
program  would  automatically  print  the  required  form  and  instructions  with  the  shopper  code  and 
shop  location  already  entered. 

Overall  Program  Effectiveness 

The  responses  on  the  subjective  rating  questions  indicated  a  response  bias.  The  ratings 
were  higher  than  expected  for  a  normal  distribution  curve.  The  ratings  fell  mostly  between  3, 4, 
and  5.  There  were  relatively  few  Is  and  2s.  These  ratings  indicated  that  the  shoppers 
approached  the  questions  with  a  positive  response  bias.  A  response  bias  could  occur  if  the 
subjects  were  influenced  to  react  a  certain  way  through  the  instructions  or  by  the  design  of  the 
experiment.  “An  interviewer  (or  trainer  in  this  case)  whose  attitude  suggests  that  some  answers 
are  more  desirable  than  others  will  get  these  answers  more  often”  (Moore  and  McCabe,  1993, 
p.252).  This  was  the  case  with  the  training  class  sessions  and  with  the  wording  on  the  rating 
scale.  Since  the  highest  rating  was  described  as  “Outstanding  (The  way  it  should  be.  Please  use 
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them!  Don’t  save  5’s  for  perfection.),”  this  helped  influence  the  shoppers  to  rate  questions  higher 
than  normal.  The  trainers  also  encouraged  the  shoppers  to  give  5s,  especially  to  recognize 
employees,  adding  to  the  response  bias.  If  an  equal  emphasis  had  been  given  to  all  ratings  on  the 
scale,  then  the  ratings  would  have  been  lower  and  more  normal. 

The  involvement  of  the  lodging  manager  could  have  an  impact  on  the  validity  of  the 
lodging  shopping  results.  Because  the  lodging  manager  had  to  be  contacted  to  assign  the 
mystery  shopper  to  a  room,  the  shopping  experience  was  no  longer  random.  In  addition,  once 
the  completed  evaluation  form  had  been  turned  in,  the  lodging  staff  could  discover  who  the 
shopper  was  by  checking  the  room  number  against  the  reservation  system.  The  other  challenge 
of  the  lodging  shops  was  the  identification  of  the  housekeeper.  Since  it  was  possible  for  the 
shopper  to  stay  overnight  without  seeing  the  housekeeper,  then  it  would  be  difficult  to  identify 
who  the  housekeeper  was  without  knowing  the  room  number  the  shopper  stayed  in  and  involving 
the  Executive  Housekeeper  to  match  that  room  to  a  staff  member.  Although,  these  challenges 
could  impact  the  quality  of  the  data,  the  results  were  still  worthwhile.  For  example,  of  the  seven 
lodging  shops,  five  identified  problems  with  worn  stained  carpet,  four  identified  that  the  spare 
pillow  and  blanket  were  not  wrapped  in  plastic  (an  AF  Innkeeper  requirement),  and  most  of  the 
written  comments  identified  old  worn  decor  and  furnishings.  However,  four  of  the  shoppers 
rated  the  housekeeping  staff  as  a  “5”  and  the  other  three  rated  housekeeping  as  a  “4.”  These 
ratings  indicate  that  the  housekeeping  staff  was  doing  a  good  job  maintaining  the  rooms  despite 
having  outdated  carpet,  decor,  and  furnishings. 

The  management  of  the  shoppers  should  also  be  improved.  There  was  no  system  in  place 
to  track  the  performance  of  each  shopper  and  identify  who  the  better  shoppers  were.  The 
performance  of  each  shopper  should  be  tracked.  The  shoppers  that  provided  the  most  complete 
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and  insightful  results  should  be  kept  and  the  shoppers  who  put  little  effort  into  the  program 
should  be  removed  from  the  schedule.  In  addition,  specific  feedback  was  not  provided  to  each 
shopper  individually.  Individualized  feedback  would  help  the  shopper  understand  where  he/she 
needed  to  improve  and  thereby  improve  the  quality  of  the  shop  data.  These  were  areas  that 
needed  to  be  handled  by  the  program  manager  as  part  of  the  program  maintenance. 
Recommendations  for  Use  of  Mystery  Shopping  Data 

The  mystery  shopping  program  results  will  only  be  effective  if  the  Services  Division 
managers  use  the  information  to  improve  their  operations.  If  the  information  just  stays  in  the 
computer,  then  there  will  not  be  any  impetus  for  improvement  and  the  money  will  be  wasted. 
For  example,  the  Services  Division  Director  could  use  the  information  to  identify  strengths  and 
weaknesses  within  his  programs.  Even  if  the  data  was  biased  in  the  positive  direction,  the 
managers  could  look  at  the  lowest  rated  areas  and  work  on  those  first.  When  these  areas  had 
been  improved,  they  could  focus  on  the  new  set  of  areas  that  were  ranked  the  lowest  and  so  on, 
creating  a  continual  improvement  process.  For  example,  employees  not  offering  additional 
information  were  identified  as  a  problem  area  (43%  of  the  employees  were  not  in  compliance). 
This  should  become  an  area  of  focus  until  it  was  improved.  By  focusing  on  this  area,  it  should 
improve  the  Division’s  performance  in  “employee  helpfulness”  and  “caring  service,”  two  of  the 
attributes  customers  ranked  in  their  top  ten.  Other  areas  such  as  identifying  the  activity  name 
over  the  phone  (100%  compliance)  and  employee  demeanor  and  greeting  (the  highest  rated  area 
with  a  4.31)  could  be  removed  from  the  evaluation  forms  and  replaced  with  other  questions  that 
better  identify  areas  needing  improvement. 

The  results  need  to  be  viewed  with  the  right  perspective.  If  one  shop  identifies  a  dirty 
bathroom  in  lodging,  it  is  not  necessarily  indicative  of  a  problem  in  all  bathrooms.  Only  after 
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several  evaluation  forms  identify  similar  problems,  can  the  problem  can  be  considered 
widespread.  The  activity  managers  need  to  closely  review  each  evaluation  form  looking  for 
these  trends.  If  a  certain  area  looks  like  a  potential  problem,  the  manager  should  validate  the 
problem  before  taking  action.  For  example,  two  of  the  seven  lodging  forms  stated  that  the  front 
desk  service  was  slow  and  not  very  friendly.  This  could  be  a  serious  problem  if  it  occurred  every 
two  out  of  seven  times,  however,  it  may  have  been  only  two  isolated  incidences  that  were  not 
part  of  a  larger  problem. 

The  commonly  rated  areas  collected  on  the  results  database  were  more  significant  from  a 
macro  perspective.  The  large  number  of  samples  increased  the  accuracy  of  the  results.  For 
example,  the  two  lodging  shops  that  identified  employees  not  wearing  their  nametags  did  not 
conclusively  identify  this  area  as  a  problem.  But,  the  55%  overall  noncompliance  with  the 
nametag  policy  did  show  a  serious  problem.  The  negative  impact  of  the  mystery  shopping’s 
small  sample  size  was  reduced  when  results  were  combined.  Large  commercial  chains  with 
standardized  procedures  could  combine  their  results  to  identify  trends  in  their  units  and  then  take 
company-wide  action  to  address  the  weak  areas.  AF  Services  as  a  whole  is  currently  unable  to 
use  this  tactic  above  the  base  level  because  of  the  decentralized  control  at  each  base  and  lack  of 
standardization.  As  AF  Services  moves  to  increase  standardization,  it  can  take  advantage  of  this 
larger  data  pool  to  identify  trends  and  make  corporate  level  decisions. 

The  compressed  time  frame  that  the  shopping  program  was  implemented  in,  combined 
with  the  large  number  of  initial  shops,  caused  several  problems.  Completing  43  shops  within  the 
initial  two  months  of  the  program’s  implementation  overloaded  the  system  and  reduced  the 
effectiveness  of  the  program.  The  program  manager  did  not  have  the  time  to  properly  review 
each  shop  as  they  came  in.  Many  of  the  blank  questions  on  the  forms  could  have  been  addressed 
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when  the  shopper  turned  the  form  in,  but  the  large  number  of  forms  coming  in  all  at  once  created 
a  backlog.  In  many  cases  the  program  manager  did  not  review  the  forms  until  after  the  shopper 
had  left. 

Another  problem  cause  by  the  large  number  of  shops  was  the  overload  of  the  recognition 
program.  Since  this  program  had  just  started,  it  took  several  weeks  to  smooth  out  the  process. 
Again,  the  time  required  by  the  program  manager  to  identify  the  employee’s  full  name,  create  the 
certificate,  and  obtain  the  necessary  approvals  was  magnified  by  the  large  number  of  employees 
being  recognized  in  such  a  short  period  of  time. 

Finally,  the  large  number  of  shops  created  a  big  demand  to  train  and  use  numerous 
shoppers.  Each  shopper  required  attention  from  the  program  manager  for  questions  and 
feedback.  A  smaller  shopper  pool  would  have  reduced  the  workload  on  the  program  manager 
and  allowed  the  shoppers  to  become  experienced  more  quickly.  These  experienced  shoppers 
would  then  reduce  the  workload  on  the  program  manager  because  their  forms  wouldn’t  need  to 
be  reviewed  as  closely. 

Recommendations  for  Future  Research 

The  area  of  mystery  shopping  is  wide-open  for  future  research.  For  this  project  in 
particular,  reactions  of  managers  and  employees  could  be  collected  on  the  effectiveness  of  the 
program.  This  information  could  be  combined  with  the  suggestions  of  the  shoppers  to  help 
improve  the  program  even  more.  The  perspective  of  the  managers  could  identify  areas  of 
improvement  most  useful  to  them. 

Future  studies  could  look  at  proving  that  mystery  shopping  actually  works.  Measuring 
financial  improvements,  customer  satisfaction  increases,  and/or  employee  productivity 
improvements  are  ways  this  could  be  accomplished. 
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Research  could  also  focus  on  the  effects  that  the  recognition  program  as  part  of  the 
shopping  program  has  on  employee  productivity  and  motivation.  The  question  that  should  be 
answered  is  -  does  the  frequency  of  the  shops  have  an  impact  on  employee  performance?  This 
question  can  be  looked  at  for  a  recognition  program,  a  disciplinary  system  or  both.  The  amount 
of  recognition  that  makes  an  impact  could  be  studied.  For  example,  employees  may  not  change 
their  behavior  for  only  a  $10  award,  but  they  might  for  a  $100  award. 

A  more  detailed  study  comparing  the  cost  of  an  internal  vs.  a  contract  program  would  be 
worth  examining.  The  expenses  of  the  internal  program  would  need  to  be  accurately  accounted 
for  to  ensure  a  good  comparison.  Even  with  an  external  contract,  the  company  must  devote 
internal  resources  to  ensure  the  program  is  developed  correctly  and  the  data  is  properly  analyzed 
and  distributed.  These  resources  need  to  be  tracked  to  ensure  an  accurate  comparison. 
Conclusions 

Based  on  the  results  of  this  project  several  conclusions  can  be  drawn.  The  best  method 
for  mystery  shopping  for  AF  bases  is  an  internally  run  program.  The  internal  program  allows  the 
Services  Division  to  focus  on  what  is  important  to  the  local  customers,  shop  activities  when  and 
where  it  wants,  and  costs  a  lot  less  than  hiring  an  outside  contractor.  The  internal  program 
appears  to  have  a  lower  shop  miss  rate  and  a  more  complete  data  set. 

Training  the  shoppers  in  advance  improves  the  shop  completion  rate  and  data  quality. 

The  shoppers  stated  that  the  training  helped  them  prepare  for  the  shop  and  answered  their 
questions.  If  no  training  had  been  provided  in  advance,  then  the  shoppers  would  have  been  left 
on  their  own  to  decide  the  best  way  to  handle  different  situations.  This  would  have  reduced  the 
quality  of  the  data  on  the  first  couple  of  shops  completed  by  each  shopper. 
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Any  other  base  could  easily  adapt  the  Wright-Patterson  Services’  mystery  shopping 
program.  The  forms,  instmctions,  shopping  guide,  training,  recruiting  materials,  and  computer 
database  could  all  be  quickly  adapted  to  fit  another  base’s  activities.  In  addition,  the  program 
could  be  expanded  to  include  other  Services’  activities  such  as  fitness  centers,  outdoor 
recreation,  and  auto  skills  center.  The  same  common  areas  could  be  rated  in  these  activities  as 
well  adding  to  the  usefulness  of  the  overall  results. 

Future  shopping  programs  should  be  started  at  a  slower  pace.  Ten  shops  within  the  first 
month  are  plenty  to  identify  and  solve  potential  system  problems  while  maintaining  a  reasonable 
workload  for  the  program  manager.  Once  the  program  is  underway  and  the  systems  are  in  place 
and  working,  more  shops  can  be  completed  each  month.  Overall  the  program  worked  well  and 
provided  the  Services  Division  with  a  valuable  tool  to  measure  their  service  quality  and  identify 
areas  for  improvement. 
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Appendix  A:  Telephone  Survey  Instruments 

Corporate  Hotel  and  Restaurant  Mystery  Shopper  Questionnaire: 

Corporate  Questionnaire  -  Company  Name: 

Contact  Name:  Title:  Date/Time: 

1 .  Do  you  use  mystery  shopping  to  evaluate  your  properties? 

la.  If  yes,  do  you  conduct  the  shops  internally  or  hire  an  external  contractor? 

Internal: 

How  do  you  select  your  shoppers? 

Do  you  train  your  shoppers  in  advance  of  shopping?  How? 

External: 

-  Does  the  contractor  train  the  shoppers  in  advance  of  shopping? 

-  What  feedback  does  the  contractor  provide?  Numeric  score?  Written 
coirunents? 

-  Are  shopping  questionnaires  and  feedback  tailored  to  fit  your  company? 

2.  What  areas  are  the  most  important  for  the  mystery  shopper  to  evaluate 
For  example:  Quality  of  product 

Customer  Service 
Sanitation 

Timeliness  of  service 
Facility  maintenance 

3.  How  often  is  each  property  shopped? 

4.  How  much  do  you  estimate  each  shop  cost? 

5.  Do  you  reward  employees  and  managers  based  on  shopping  results? 

5a.  If  yes,  how  do  you  reward  them? 

6.  What  problems  do  you  most  often  encounter  with  the  program? 

7.  Would  you  mind  sending  me  a  sample  questionnaire? 

8.  Company  Address: 
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Commercial  Mystery  Shopping  Company  Questionnaire 
Company  Name;  Date/Time: 

Contact  Name:  Title: 

1.  How  do  you  recruit  new  shoppers? 

2.  On  average  how  much  do  you  charge  per  shop  above  the  cost  of  the  food/services 
utilized? 

2a.  For  a  restaurant? 

2b.  For  a  hotel? 


3.  How  often  do  you  recommend  a  restaurant  be  shopped?  Hotel? 


4.  How  do  you  train  your  shoppers  before  they  start  shopping? 


5.  Do  you  compensate  shoppers  for  more  than  the  free  meal  or  hotel  room?  How  much? 


6.  Do  you  use  standard  shopping  questionnaires  or  tailor  them  to  each  company  being 
shopped? 


7.  What  areas  are  the  most  important  for  the  mystery  shopper  to  evaluate? 
For  example:  Quality  of  product 
Customer  Service 
Sanitation 

Timeliness  of  service 
Facility  maintenance 


8.  What  problems  do  you  most  often  encounter  with  mystery  shopping? 

9.  Would  you  mind  sending  me  sample  hotel  and  restaurant  questionnaires? 


APPENDIX  B:  Mystery  Shopper  Program  Elements 
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Appendix  B 

Mystery  Shopper  Program  Elements 
Create  shopper  recruiting  plan 

-  Identify  where  to  recruit  potential  shoppers  (Newcomer’s  Orientation?  Base 
Exchange/Commissary?  Advertise  in  facilities?  Base  Paper?) 

-  Identify  correct  number  of  shoppers  to  maintain  an  adequate  pool  of  shoppers 

-  Important  to  recrait  cross-section  of  customer  base  to  represent  all  consumer  groups. 
Identify  the  best  shopper  profiles  for  each  activity. 

Create  marketing  materials  for  recruiting  and  promoting  program 

-  Logo  and  mystery  shopper  program  theme 

-  Pamphlets  to  hand  out  discussing  the  program 

-  On-going  publicity  and  recruiting  activities 

-  Article  for  base  paper  publicizing  the  program 

Create  payment  procedures 

-  Identify  reimbursement  amount  of  shop  up  to  certain  dollar  value 

-  Identify  compensation  for  mystery  shopper 

-  Reimbursement  only  granted  after  a  completed  report  is  turned  in 

-  Work  out  details  for  appropriated  fund  facilities 

-  Work  with  NAF  Accounting  Office 

Design  activity  specific  questionnaires 

-  Work  with  activity  managers,  marketing,  and  training  to  identify  the  most  important 
areas  to  focus  on  -  it  is  important  to  gather  similar  data  to  identify  squadron  wide 
trends 

-  Benchmark  off  of  commercially  available  forms 
Create  shopper  training  materials 

-  Create  training  course  to  teach  future  shoppers  how  to  remain  anonymous,  what  items 
to  look  for,  techniques  to  use  in  observing,  and  how  to  complete  the  questionnaire 
afterwards 

-  Create  scenarios  for  trainees  to  evaluate  and  critique  during  training 

-  Create  the  Services  Guide  to  Mystery  Shopping  that  shoppers  can  keep  and  refer  to 
after  training 

Design  activity  specific  schedules  for  shopping 

-  Work  with  activity  managers  to  identify  critical  times  for  shopping 

-  Identify  proper  firequency  and  times  of  shops  for  accurate  feedback 

Create  employee  recognition  program  to  correspond  with  program 

-  Identify  criteria  for  selecting  award  recipients 
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-  Set  award  amount 

-  Identify  presentation  procedure  (at  work.  Commander’s  call,  etc.) 

-  Create  certificate  or  other  document  to  accompany  monetary  or  other  award. 

-  Identify  list  of  possible  awards  for  employees  to  recommend  for  Air  Force  level 

Administrative  requirements: 

-  Create  budget  for  shopper  program  based  on  potential  compensation  and  frequency  of 
shops 

-  Train  managers  on  program 

-  Advertise  program  to  employees  (emphasize  positive  aspects) 

-  Collect  copies  of  computer  files  of  all  aspects  of  the  program 

-  Create  binder  of  all  mystery  shopper  materials 

-  Work  with  the  Services  Agency  to  fund  duplication  and  distribution  of  materials 

-  Create  computer  program  to  tabulate  results  and  print  results 

-  Create  computer  database  to  track  shoppers  (dates,  locations,  effectiveness) 

Program  follow-up: 

-  Provide  feedback  to  shoppers  on  proper  form  completion 

-  Possible  refresher  training  to  get  feedback  on  program  and  reinforce  concepts 
Continuous  recruiting  to  ensure  shoppers  are  new  and  prevent  routine  shoppers  from 
being  identified 


APPENDIX  C:  Project  Timeline 
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Mystery  Shopper  Timeline  (1998): 

Mar  16  -  Program  planning  session 

Mar  17  -  May  10 

-  Design  draft  questionnaires 

-  Contact  commercial  companies  for  information  on  programs 

-  Design  publicity  and  recruiting  materials 

-  Decide  payment  procedures 

May  10 -May  31 

-  Write  Mystery  Shopper  pamphlet 

-  Design  Training  Program 

-  Design  recognition  program 

-  Recmit  Shoppers  (May  -  July,  Schedule  for  training  in  Aug) 

Jun  1  -  Jul  31 

-  Get  feedback  from  managers  on  questionnaires 

-  Inform  managers  about  the  program 

-  Publicize  program  to  employees 

-  Create  Services’  Guide  to  Mystery  Shopping 

-  Coordinate  contract  through  legal  if  necessary 

Aug  1  -  Aug  8 

-  Train  shoppers 

-  Increase  publicity  within  facilities  of  program  benefits  and  purpose 
Aug  9 -Aug  31 

-  Begin  shopping  at  all  facilities  based  on  schedule 

-  Add  extra  shops  for  the  purpose  of  the  study  at  a  couple  locations  (10  lodging  and  40 
foodservice  shops  total) 

Sep  1  -  Oct  30 

-  Obtain  feedback  from  shoppers  on  program 

-  Make  necessary  adjustments  to  program  based  on  feedback 

-  Create  program  to  track  data  and  shoppers 

-  Ensure  recruiting  continues 

-  Schedule  second  training  session  (quarterly?) 

-  Write  research  project  that  will  impress  committee 

Nov  1  -  Nov  14 

-  Defend  final  research  project 

-  Celebrate  (a  little) 


Nov  15 -Dec  5 

-  Make  necessary  changes  to  proposal 

-  Obtain  final  approval  from  committee 

-  Bind  project 

-  Celebrate  (a  lot) 

Graduate!! 


APPENDIX  D:  Tri-fold  Informational  Recruiting  Brochure 
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%^“-;f%bP^iD^YSTEFi¥  SjHOSPER  BUDGBF^/f 
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Activity  Name 

Frequency 

Budgeted 
Cost  Per 
Shop 

Gift 

Certificate 

Value 

Budgeted  Number  of 
Annual  Shops 

Cost  Completed 

Actual 

Reimburse¬ 

ment 

Total 

Actual 

Cost 

Community  Support  Flight 

Skills  Development  Center 

Quarterly 

$10 

$10 

$80 

Auto  Skills  Center 

Quarterly 

$20 

$10 

$120 

Rod  and  Gun  Club 

Quarterly 

$0 

$10 

$40 

Outdoor  Recreation 

Quarterly 

$15 

$10 

$100 

Aquatics 

Bi-Monthly 

$5 

$10 

$90 

Community  Support  Totals: 

$430  0 

$0.00 

$0.00 

Family  Member  Support  Flight 

New  Horizon  CDC 

Quarterly 

$0 

$10 

$40 

Kittyhawk  CDC 

Quarterly 

$0 

$10 

$40 

Page  Manor  CDC 

Quarterly 

$0 

$10 

$40 

Wright  Care  CDC 

Quarterly 

$0 

$10 

$40 

Preschool 

Quarterly 

$0 

$10 

$40 

Tots  In  Blue 

Quarterly 

$0 

$10 

$40 

Youth  /  CAC 

Quarterly 

$0 

$10 

$40 

Family  Member  Totals: 

$280  0 

$0.00 

$0.00 

Combat  Support  Flight 

Pitsenbarger  Dining  Facility 

Monthly 

$5 

$10 

$180  2 

$4.75 

$24.75 

Lodging 

Monthly 

$17 

$10 

$324  7 

$107.00 

$177.00 

Library 

Quarterly 

$0 

$10 

$40 

Dodge  Gym 

Bi-Monthly 

$5 

$10 

$90 

Hangar  22  Fitness  Center 

Bi-Monthly 

$5 

$10 

$90 

Jarvis  Gym 

Bi-Monthly 

$5 

$10 

$90 

Health  Club  Area  A 

Bi-Monthly 

$5 

$10 

$90 

Health  Club  Area  B 

Bi-Monthly 

$5 

$10 

$90 

Combat  Support  Totals: 

$994  9 

$111.75 

$201.75 

Mystery  Shopper  Budget 
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Budgeted 

Gift 

Budgeted 

Number  of 

Actual 

Total 

Cost  Per 

Certificate 

Annual 

Shops 

Reimburse¬ 

Actual 

Activity  Name 

Frequency 

Shop 

Value 

Cost 

Completed 

ment 

Cost 

Business  Operations  Flight 

Officer's  Club 

Monthly 

$15 

$10 

$300 

4 

$41.85 

$81.85 

The  Wright  Place 

Monthly 

$0 

$0 

$0 

Flywright  Club 

Base  Restaurants 

Monthly 

$10 

$10 

$240 

2 

$14.95 

$34.95 

(Table  Service) 

Bi-Monthly 

$15 

$10 

$150 

2 

$9.75 

$29.75 

Base  Cafeterias 

Monthly 

$10 

$10 

$240 

8 

$40.60 

$120.60 

Base  Canteens 

Bi-Weekly 

$5 

$10 

$360 

13 

$50.02 

$180.02 

Twin  Base  Golf  Course* 

Bi-Monthly 

$10 

$10 

$120 

1 

$8.35 

$18.35 

Prairie  Trace  Golf  Course' 

'  Bi-Monthly 

$10 

$10 

$120 

2 

$3.70 

$23.70 

Kittyhawk  Bowling  Center’ 

'  Bi-Monthly 

$15 

$10 

$150 

2 

$9.30 

$29.30 

Civilian  Recreation 

Quarterly 

$0 

$10 

$40 

Aero  Club 

Quarterly 

$0 

$10 

$40 

Veterinary  Clinic 

Quarterly 

$0 

$10 

$40 

Business  Operations  Totals: 

$1,800 

34 

$178.52 

$518.52 

Other 

Unused  Gift  Certificates 
(Subtracted  from  Total) 

Marketing  Expenses 

Human  Resoures  Office 

Quarterly  $0 

$10  $40 

$30.00 

Other  Total: 

$40 

0 

$0.00 

-$30.00 

OVERALL  TOTALS: 

$3,544 

43 

$290,27 

$690,27 

KEY  ASSUMPTIONS: 

Frequency _ Shops  per  year 


Annually  1 

Bi-Annually  2 

Quarterly  4 

Bi-Monthly  6 

Monthly  12 

Bi-Weekly  24 

Weekly  52 

Notes: 


*  Seasonal  Activities  such  as  Golf,  Bowling  and  Aquatics  are  listed  as  Bi-Monthly  with  6  shops  per  year. 
These  shops  will  probably  take  place  during  the  peak  season,  with  few  or  no  shops  during  off-season. 


Mystery  Shopper  Budget 


11/30/98 
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APPENDIX  F:  Mystery  Shopping  Schedule 


Mystery  Shopping  Schedule 


1100-1300  M-F  Smith,  Kerry  Yes 


tember  |Canteen#24  Lunch  31-Auq-98  9-Sep-98  1100- 1300  M-F  Williams.  Jeanette 
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APPENDIX  G:  88*  Services  Division  1998  Customer  Satisfaction  Survey  Results 


Total  Survey  Respondents 


3  {0%I 


2  {o%l; 


I  Enlisted 


I  Officer 


H  Civilian 


H  Retirees 


I  Reserv'es 


ARTS 


I  Contractors 


2  (0%) 


50 


100 


150 


200 


Importance  of  Attributes  in  Rank  Order 


Score 

(Mean) 

Count 


92%  (4.60) 
92%  (4.60) 
92%  (4.59) 
90%  (4.52) 
90%  (4.48) 
89%  (4.47) 
89%  (4.46) 
88%  (4.38) 
85%  (4,24) 
84%  (4.19) 
81%  (4,07! 
81%  (4.06) 
81%  (4.06) 
78%  (3.90) 
66%  (3.28) 


i6o 


I  Employ  Courtesy 
I  Service  Correct 
I  FacU  Cleanliness 
I  Employ  helpfull 
I  Equip  Condition 
Equip  Cleanliness 
I  Service  Promptness 
I  Product  Knowledge 
I  Caring  Service 
I  Understanding 
I  Employ  Appearance 
I  Individual  Service 
I  FacU  Appearance 
I  Accurate  Info 
I  Ad  Material  Quality 


Overall  85%  (4.26):  (Total) 


Satisfaction  With  Attributes  in  Rank  Order 


80%  (3.98)  625j 


74%  (3.69)  615  — — 

604 

582 

610 

71%  587  I 

0  2b  4o  do  80 


I  Facil  Cleanliness 
I  Facil  Appearance 
I  Equip  Cleanliness 
I  Employ  Appearance 
I  Employ  Courtesy 
Equip  Condition 
m  Service  Correct 
I  Product  Knowledge 
I  Employ  heipfuil 
I  Caring  Seiv’lce 
I  Service  Promptness 
I  Understanding 
H  Accurate  Info 
I  Individual  Service 
I  Ad  Material  Quality 


Overall  76%  (  3.79):  (Total) 


Overall  Installation  Grade  in  Meeting  Needs 


I  Very  Best 
I  Excellent 
I  Very  Good 
I  Quite  Good 
I  Fairly  Good 

indiferent 

I  Not  Very  Good 
I  Not  Good  at  ail 

I  ^t)or 

I  Terrible 
I  Very  Worst 


Scale  Value  Mean  =  75.89  CTotal) 


APPENDIX  H:  Example  Evaluation  Forms 


All  ratings  are  based  on  a  5-point  scale:  1=Poor,  2=Fair,  3=Satisfactory,  4=Good,  5= 
Write  comments  explaining  all  ratings  2  or  lower  and  all  NO  answers.  Write  on  back 


Was  the  phone  answered  in  three  rings? 


Did  the  employee  answering  the  phone  identify  himself/herself  voluntarily? 


Name  of  employee  answering  the  phone: 


Did  this  person  identify  their  work  center  (i.e.  front  desk  or  reservations)? 


Did  this  person  have  a  friendly  demeanor  and  extend  a  greeting?  Please  rate.  _ 


Were  directions  or  additional  information  offered? 


If  you  were  placed  on  hold,  how  long  did  you  have  to  wait  (in  seconds)?  _ 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness’ 


Comments: 


Outstanding. 

if  necessa 


mmm 


□yes  Gno 


□yes  Dno 


□yes  Gno 


□yes  Gno 


□yes  Gno 


Seconds 


wmmmmmmmm 


Rate  the  exterior  appearance  of  the  facility  (cleanliness,  maintenance,  and  landscaping 


Rate  the  interior  appearance  of  lobby,  hallways,  and  stairwells  (cleanliness,  maintenance,  etc. 


Rate  the  lobby  atmosphere  (decor,  lighting,  temperature,  seating,  etc. 


Rate  the  condition  and  cleanliness  of  the  public  restrooms  □  Men’s  □  Women's 


Was  the  laundry  room  convenient,  clean,  and  comfortable,  with  folding  table  and  hanging  rack? 

u 

YES 

O 

NO 

Were  the  ice  machines  conveniently  located  and  in  good  working  order? 

n 

YES 

n 

NO 

Were  the  parking  areas  clean  with  adequate  lighting? 

n 

YES 

n 

NO 

Were  directional  signs  to  your  room  attractive  and  easy  to  follow? 

n 

YES 

n 

NO 

Name  of  front  desk  employee  who  checked  you  in: 


Description  (sex,  height,  hair  color,  approx  age): 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 


Did  the  clerk  ask  if  you  needed  any  information  about  the  base  or  local  area? 


Name  of  front  desk  employee  who  checked  you  out: 


□yes  Gno 


I  Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 

i _ 

Was  your  bill  accurate? 

□yes  Gno 

Were  you  quickly  greeted  upon  arriyal  at  the  front  desk  each  time  and  made  to  feel  welcome? 

□yes  Gno 

Were  staff  members  in  proper  uniforms  with  nametags?  Were  they  neat  and  clean? 

□yes  Gno 

Did  the  clerk{s)  provide  friendly  and  helpful  service?  Please  rate. 

□yes  Gno 

Were  staff  members  knowledgeable  about  their  jobs,  the  base  and  local  area?  Please  rate. 

□yes  Gno 

Was  the  check-in/out  process  quick  and  efficient?  (1  min.  for  check-in,  3  min.  for  check-out) 

□yes  Gno 
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Was  the  carpet  clean  and  free  of  stains  and  wear  marks? 


Was  the  furniture,  counters,  ledges,  etc.  free  of  dust? 


Was  the  bathroom  clean?  Did  it  smell  fresh?  Was  the  toilet  clean  inside  and  out? 


Was  the  shower  /  bathtub  clean?  (free  of  soap  scum  and  mold?  clean  shower  curtain,  etc.?' 


Were  the  sink  and  vanity/mirrors  clean?  (including  so^  dish  and  counter  if  applicable 


Were  the  microwave  and  refrigerator  clean?  (Check  the  inside  door  and  ceilin 


Were  the  towels  (two  sets)  and  linens  clean  and  in  good  condition? 


Was  the  bed  well  made? 


Were  amenities  provided?  (Shampoo  with  conditioner,  mouthwash,  lotion  and  shoe  mitt. 


Were  quality  brand  facial  and  toilet  tissue  provided?  (2  ply  is  standard 


Were  deodorant  soap  and  facial  soap  provided?  (Quality  commercial  product,  1.25  oz. 


Was  the  housekeeper  wearing  a  uniform  with  a  name  tag?  Was  it  neat  and  clean? _ 


Name  of  your  housekeeper: 


Rate  the  performance  of  your  housekeeper  (room  cleanliness,  stocking,  and  courtes 


Comments: 


□yes 

□  no 

□yes 

□no 

□yes 

□  no 

□yes 

□no 

□yes 

□no 

□yes 

□  no 

□yes 

□no 

□yes 

□no 

□yes 

□no 

□yes 

□  no 

□yes 

□no 

□yes 

□no 

mmwmmmmB 


Was  the  quest  information  book  in  good  condition? 


Did  the  quest  information  book  provide  useful  and  accurate  information? 


Were  the  TV,  remote  control,  VCR,  phone  and  clock  radio  working  properly? 


Were  clear  instructions  provided  to  operate  all  equipment  in  the  room? 


Was  there  an  iron  and  ironing  board  provided  and  in  good  condition? 


Were  ten  hangers  provided?  Were  at  least  4  of  them  skirt  hangers? 


Did  the  room  have  adequate  lighting? 


Was  everything  in  your  room  in  working  order? 


Did  the  furnishings  and  decor  compliment  each  other? 


Were  the  furnishings  in  good  condition  and  well  placed? 


Were  the  drapes  in  good  condition?  Did  they  block  all  outside  light  when  closed? 


Were  there  working  light  bulbs  in  every  lamp  including  3-way  bulbs  in  3-way  lamps? 


Was  the  room  air  quality  good?  Did  the  bathroom  exhaust  fan  work  properly? 


Were  the  spare  blankets  and  pillows  wrapped  in  plastic? 


Was  the  room  stocked  with  resale  items?  Was  it  a  good  selection? 


Comments  (including  resale  suggestions): 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes 


□yes  □no 


□yes  □no 


□yes  □no 


Rate  the  overall  room  quali 


Rate  the  overall  quality  of  service  provided  by  the  lodging  staff 


Rate  the  perceived  value  for  the  dollar 


Rate  the  overall  cleanliness 


Rate  your  overall  lodging  experience 


Based  on  THIS  experience,  would  you  return? 


Overall  comments: 


□yes  □no 
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SSYSTERY  SHOPPER  EVALU ATI^)N  FORiyi  -  OFFICERS’  CLm  JABlE  SERVICE 


Day  of  Week: 


Shopper: 

Location: 

Officers'  Club 

rC^sfc . 


All  ratings  are  based  on  a  5-point  scale:  1=Poor,  2=Fair,  3=Satisfactory,  4=Good,  5=Outstanding. 
Write  comments  explaining  all  ratings  2  or  lower  and  all  NO  answers.  Write  on  back  if  necessa 


Was  the  phone  answered  in  three  rings? 


Did  the  person  answering  the  telephone  identify  himself/herself  voluntarily? 


Name  of  person  answering  the  phone: 


Did  this  person  identify  the  business  name? 


Did  this  person  have  a  friendly  demeanor  and  extend  a  greeting?  Please  rate. 


If  you  were  placed  on  hold,  how  long  did  you  have  to  wait  (In  seconds)? _ 


Did  this  person  offer  directions  or  additional  information? 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 


Comments: 


□yes 


□yes 


□yes 


□yes 


NO 


0 


Seconds 


□  no 


Rate  the  exterior  appearance  of  the  facility  (cleanliness,  maintenance,  and  landscaping 


Rate  the  interior  appearance  of  the  facility  (cleanliness,  maintenance,  etc. 


Rate  the  atmosphere  (decor,  lighting,  temperature,  music,  etc. 


Rate  the  condition  and  cleanliness  of  the  public  restrooms  □  Men's  □  Women's 


Was  your  table  clean  and  attractive  including  chairs,  utensils,  condiments,  etc.?  Dyes 


Was  the  floor  and  carpet  clean  and  free  of  stains?  □yes 


Comments: 


. 


Server's  name: 


Description  (sex,  height,  hair  color,  other): 


Rate  the  overall  performance  of  the  server  (consider  courtesy,  accuracy  and  helpfulness 


RAi»tafi- 


Were  you  quickly  greeted  upon  arrival  and  made  to  feel  welcome? 

□yes 

□  no 

Was  the  server  wearing  a  uniform  with  a  name  tag?  Was  it  neat  and  clean? 

[■OSHllSH 

Did  the  server  provide  friendly  and  helpful  service?  Please  rate. 

□yes 

□no 

Were  the  menus  clean  and  in  good  condition? 

□yes 

□  no 

Was  your  server  knowledgeable  of  the  menu  and  specials  (if  any)?  Please  rate. 

□yes 

□no 

Did  the  server  provide  any  menu  selection  suggestions  (appetizer,  dessert,  etc.)? 

□yes 

□no 

How  long  did  it  take  for  your  drinks  to  arrive  after  ordering? 

Minutes 

How  long  did  it  take  for  your  entrees  to  arrive  after  ordering? 

Minutes 

Was  your  food  the  appropriate  temperature  upon  arrival? 

□yes 

□  no 

Was  your  table  and  others  around  you  properly  bussed? 

□yes 

□  no 

Were  after  meal  drinks  suggested  by  the  server? 

□yes 

□  no 

Was  your  guest  check  accurate? 

□yes 

□  no 
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wmmim. 


Host  or  Additional  Server's  Name: 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 

i: _ 

Did  the  anyone  from  the  staff  check  your  club  membership? 

□yes  □no 

Did  you  observe  a  manager?  Did  he/she  appear  to  be  involved  in  running  the  restaurant? 

□  yes  □no 

Did  anyone  from  the  staff  thank  you  for  coming? 

□  yes  □no 

Name  of  your  bartender  /  server: 


Description  (sex,  height,  hair  color,  other): 


Rate  the  overall  performance  of  the  server  (consider  courtesy,  accuracy  and  helpfulness 


Were  you  quickly  qreeted  by  the  bartender/server? 


Was  the  bartender  /  server  wearing  a  uniform  with  a  name  tag?  Was  it  neat  and  clean? 


Did  the  bartender  /  server  smile  and  make  eye  contact? 


Did  the  bartender  /  server  ask  to  see  any  identification? 


Was  your  bartender  /  server  knowledgeable  of  drink  types  and  the  bar  menu? 


Did  the  bartender  /  server  make  any  suggestions  (drink,  appetizer,  etc.)? 


Was  the  bar /lounge  area  clean? 


Did  you  receive  your  drink(s)  in  a  timely  manner? 


Was  your  bar  bill  accurate? 


Comments: 


□yes 


□yes 


□yes 


□yes 


□yes 


Rate  the  overall  quality  of  service  provided  by  the  staff 


Rate  the  perceived  value  for  the  dollar 


Rate  the  overall  cleanliness 


Rate  the  overall  Officer's  Club  experience 


Based  on  THIS  experience,  would  you  return? 


Overall  comments: 


□yes  □no 
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APPENDIX  I:  Common  Rated  Areas  on  All  Evaluation  Forms 


Common  areas  rated  on  all  mystery  shopping  questionnaires: 


Pre-Visit  Call: 

Phone  answered  in  three  rings  Yes/No 

Employee  identify  themselves  Yes/No 

Name  of  employee  * 

Employee  identify  business  activity  Yes/No 

Friendly  demeanor  and  extend  a  greeting  Rate 

Offer  additional  information  or  directions  Yes/No 

Overall  performance  of  this  employee  Rate* 

Facility  Evaluation; 

Exterior  of  facility  Rate 

Interior  of  facility  Rate 

Atmosphere  Rate 

Restroom  cleanliness  Rate 


Customer  Service  (direct  employee  interaction): 


Name  of  employee  * 

Description  (to  ensure  proper  ID)  * 

Overall  performance  of  employee  Rate* 

Quickly  greeted  Yes/No 

Friendly  and  helpful  service  Rate 

Knowledgeable  staff  Rate 

Neat  and  clean  uniforms  with  name  tags  Yes/No 

Accurate  bill  Yes/No 

Overall  Ratings: 

Product  quality  (food,  fitness  equipment,  etc)  Rate 

Quality  of  service  by  staff  Rate 

Perceived  value  for  the  dollar  Rate 

Cleanliness  Rate 

Overall  experience  Rate 


•  Items  with  an  asterisk  “*”  will  be  used  to  determine  individual  employee  recognition. 

•  There  are  several  categories  that  apply  to  many  activities,  but  not  all  of  them.  For  example 
suggestive  selling  and  visible  management.  These  items  can  be  tracked  separately. 

•  All  ratings  are  on  a  5-point  scale 


APPENDIX  J:  Example  Shopping  Instructions 
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Mystery  Shopping  Instructions  -  Lodging 


General  Instructions: 

Shop  must  be  completed  between: _ and _ .  Make  your  reservation  right  away  to 

ensure  room  availability.  Return  your  completed  form  to  Services  Marketing  within  48  hours  after  the  shop  or 
the  next  business  day  if  shop  falls  on  a  Friday  or  Saturday.  You  will  be  reimbursed  for  one  night  stay  in 
lodging  (maximum  reimbursement  is  $14.00).  K  the  form  is  completed  properly,  on  time,  and  legibly,  you  will 
receive  $10  in  Services’  gift  certificates  to  use  in  any  of  our  activities. 


Read  the  questionnaire  thoroughly  before  shopping.  Think  about  what  you  will  need  to  do  to  answer  each 
question. 

Please  write  legibly  and  complete  the  form  with  black  or  blue  ink  pen  only. 

Check  “Yes”  and  “No”  boxes  clearly.  Only  check  one  for  each  question. 

For  rated  questions,  use  a  whole  number  between  1  and  5.  Consider  the  following  scale  when  rating  questions: 

5  =  Outstanding  (The  way  it  should  be.  Please  use  them!  Don’t  save  5’s  for  perfection.) 

4  =  Good 

3  =  Satisfactory  (Nothing  special,  plain,  ordinary) 

2  =  Fair 

1  =  Poor  (Offensive,  negligent,  hazardous) 

You  must  write  comments  explaining  all  “No”  answers  and  all  ratings  2  or  lower 
Make  sure  the  ratings  agree  with  the  written  comments. 

Keep  your  itemized  receipt  and  turn  it  in  with  the  completed  form!  Telephone  charges  and  sundry  charges 
(resale  items  in  the  rooms)  are  not  reimbursed  and  are  the  responsibility  of  the  shopper  to  pay. 

Make  sure  you  obtain  the  employees’  names  (including  the  reservationist). 

Coupons  received  for  mystery  shopping  cannot  be  used  to  pay  for  a  mystery  shop. 

Please  make  all  comments  constructive  -  avoid  writing  your  opinions  too  harshly. 

Refer  to  your  “Services’  Guide  to  Mystery  Shopping”  or  call  Marketing  if  you  have  a  question. 


Specific  Instructions: 

When  making  your  reservation,  tell  the  reservationist  that  you  are  on  orders  and  will  be  staying  only  one  night. 
Provide  your  real  name  and  social  security  number  (they  will  check  your  ID  when  you  check-in).  This  will 
allow  you  to  complete  the  reservation  process.  Please  call  between  7:30  and  5:00.  The  reservation  number  is 
787-3810.  This  is  a  good  time  to  get  directions  if  needed. 

Once  you  have  made  your  reservation,  you  need  to  call  Mr.  Ray  Zimmerschied,  the  Lodging  Manager,  at  257- 
2928.  Tell  him  you  are  a  mystery  shopper  and  give  him  your  name  and  confirmation  number.  He  will  change 
your  reservation  in  the  computer  so  you  can  check-in  without  orders.  He  will  also  assign  you  to  the  room  type 
(VAQ,  VOQ,  etc.)  that  is  next  in  line  to  be  evaluated.  Please  check-in  after  1400. 

Sit  in  the  lobby  and  look  at  information  and  services  provided  (visit  the  lobby  restrooms). 

Evaluate  the  interior  and  exterior  of  the  lobby  and  of  the  building  you  are  staying  in. 

Visit  the  laundry  and  ice  machine  areas  in  your  building. 

Call  the  front  desk  and  ask  one  or  more  of  the  following  questions  to  rate  employee  knowledge: 

Hours  of  operation  and  directions  to  the  BX,  Commissary,  or  any  Services  facility 
Shuttle  bus  schedule  and  location  of  stops  or  eating  establishments  in  the  local  area 
It  is  essential  that  you  obtain  the  name  of  your  housekeeper.  If  you  do  not  see  him  or  her,  call  the  Executive 
Housekeeper’s  Office  (after  you  have  checked-out)  at  257-3464.  Identify  yourself  as  a  mystery  shopper  and 
that  you  need  the  name  of  the  housekeeper  for  the  room  you  stayed  in.  Thanks  and  have  fun  shopping! 
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Mystery  Shopping  Instructions  -  Officers’  Club  Table  Dining 


General  Instructions: 

Location:  Building  800  Area  A 

Shop  must  be  completed  by: _ .  Shop  any  day  between _  and _ .  Shop  any  time 

between _ and _ .  Return  your  completed  form  to  Services  Marketing  within  48  hours  or  by  the 

completion  date  -  whichever  is  sooner.  You  will  be  reimbursed  for  your  meal  expenses  (maximum 
reimbursement  is  $20.00).  If  the  form  is  completed  properly,  on  time,  and  legibly,  you  will  receive  $10  in 
Services’  gift  certificates  to  use  in  any  of  our  activities. 

Read  the  questionnaire  thoroughly  before  shopping.  Think  about  what  you  will  need  to  do  to  answer  each 
question. 

Please  write  legibly  and  complete  the  form  with  black  or  blue  ink  pen  only. 

Check  “Yes”  and  “No”  boxes  clearly.  Only  check  one  for  each  question. 

For  rated  questions,  use  a  whole  number  between  1  and  5.  Consider  the  following  scale  when  rating  questions: 

5  =  Outstanding  (The  way  it  should  be.  Please  use  them!  Don’t  save  5’s  for  perfection.) 

4  =  Good 

3  =  Satisfactory  (Nothing  special,  plain,  ordinary) 

2  =  Fair 

1  =  Poor  (Offensive,  negligent,  hazardous) 

You  must  write  comments  explaining  all  “No”  answers  and  all  ratings  2  or  lower 
Make  sure  the  ratings  agree  with  the  written  comments. 

Keep  your  itemized  receipt  and  turn  it  in  with  the  completed  form!  Ask  for  receipt  if  one  is  not  provided. 
Alcoholic  beverages  wiU  NOT  be  reimbursed  and  must  NOT  be  included  on  the  receipt. 

Make  sure  you  obtain  the  employees’  names  including  the  employee  answering  the  phone. 

Coupons  received  for  mystery  shopping  cannot  be  used  to  pay  for  a  mystery  shop. 

Please  make  all  comments  constructive  -  avoid  writing  your  opinions  too  harshly. 

Refer  to  your  “Services’  Guide  to  Mystery  Shopping”  or  call  Marketing  if  you  have  a  question. 

Specific  Instructions: 

Call  before  you  visit  the  restaurant.  The  number  is  257-9762.  Ask  the  employee  what  the  special  is,  when  they 
are  open,  and  get  directions. 

Visit  the  public  restrooms.  Evaluate  for  cleanliness  and  stocking  of  soap  and  paper. 

Evaluate  the  interior  and  exterior  of  the  Club. 

Purchase  a  full  meal  including  beverage.  Purchase  an  appetizer  or  dessert. 

Look  at  any  salad  bar,  taco  line,  deli  sandwich  line,  potato  bar,  etc.  even  if  you  don’t  order  from  them. 

Ask  your  server  about  the  menu  and  specials  to  rate  employee  knowledge. 

Evaluate  the  general  service  at  other  tables  and  how  well  the  tables  are  bussed. 

It  is  essential  that  you  obtain  the  names  of  your  server  and  bartender,  ask  if  necessary. 

Don’t  forget  to  tip!  15%  is  a  good  rule  (Reference  your  Guide  to  Mystery  Shopping). 

Visit  Wings  (the  O’Club  bar)  and  purchase  a  drink.  Alcohol  will  not  be  reimbursed,  you  can  buy  a  non¬ 
alcoholic  drink,  but  you  are  encouraged  to  try  an  alcoholic  drink  and  let  us  know  how  we  are  doing. 

Thanks  and  have  fun  shopping! 
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Introduction  -  What  is  Mystery  Shopping? 

Welcome  to  the  88*  Services  Division’s  mystery  shopping  program.  Mystery  shoppers  secretly 
evaluate  Services’  facilities  while  posing  as  regular  customers.  The  goal  of  this  program  is  to  obtain 
unbiased  feedback  from  our  customers  about  their  service  experience.  In  addition,  we  want  to  reward  our 
employees  for  meeting  our  high  quality  standards  and  providing  outstanding  customer  service  to  you,  our 
customer.  This  guide  is  designed  to  supplement  the  formal  classroom  training  and  serve  to  answer  your 
questions  regarding  the  program.  Please  review  this  guide  before  each  shop  to  refresh  your  memory  on  the 
requirements  and  techniques  used. 

Mystery  shopping  assignments  (referred  to  as  “shop”  or  “shops”  in  this  guide)  are  scheduled  by  the 
88*  Services  Marketing  Department.  Ms.  Sonya  Greene,  Events  and  Sponsorship  Coordinator,  is  the 
primary  point  of  contact  for  running  the  program.  She  will  handle  on-going  shopper  recruiting,  scheduling 
training,  matching  shoppers  to  shops  and  collecting  completed  shops,  and  handling  any  questions 
concerning  the  program.  She  will  be  your  main  contact  as  a  Services’  mystery  shopper. 

Once  you  have  completed  mystery  shopping  training,  your  information  will  be  kept  in  our  shopper 
database  for  future  shopping  opportunities.  You  will  be  contacted  by  telephone  or  e-mail  when  you  are 
selected  for  a  shop.  If  you  are  available  for  the  shop,  you  will  need  to  pick-up  the  shopping  form  and 
specific  shopping  instructions  (see  Attachment  1  for  sample  instructions)  by  the  agreed  upon  date  at  the 
Services  Marketing  Office  in  Bldg.  70  (Enter  door  with  “MWRS”  awning)  room  222  .  The  address  is  5215 
Thurlow  Street,  Suite  2,  located  in  Area  C.  If  you  need  directions  please  call  (937)  257-8220. 

How  Shoppers  Are  Selected  For  Shops 

Shoppers  are  matched  with  each  shop  based  on  a  number  of  items.  The  first  and  most  important  is 
the  shopper  demographic  information.  We  want  to  make  sure  that  the  shopper  matches  as  closely  as 
possible  to  the  profile  of  the  business’  target  customer.  For  example,  we  want  golfers  to  shop  our  golf 
course  and  bowlers  to  shop  the  bowling  center  and  so  on.  Another  consideration  is  frequency.  We  want  as 
many  new  eyes  to  look  at  our  activities  as  possible.  For  example,  a  shopper  who  hasn’t  been  to  the  golf 
course  is  more  likely  to  be  selected  than  someone  who  has  already  shopped  there.  Shopper  performance  is 
also  a  factor.  The  shoppers  who  take  the  program  seriously  and  consistently  provide  complete,  legible,  and 
timely  evaluations  are  more  likely  to  receive  additional  shopping  opportunities. 

When  your  demographic  profile  matches  an  upcoming  shop,  we  will  call  and  provide  you  with  only 
the  dates  and  times  of  the  shop  to  allow  you  to  make  the  decision  to  accept  or  decline  (except  lodging). 
Once  you  have  accepted  the  shop,  we  will  provide  you  with  the  location  and  other  pertinent  information 
concerning  the  visit.  This  procedure  ensures  that  all  of  our  activities  are  shopped  when  needed,  and  by  a 
shopper  with  the  right  customer  profile. 

Before  Shopping 

Always  read  over  each  question  on  the  evaluation  form  just  before  you  shop.  The  questions  will  be 
fresh  on  your  mind  and  you  can  plan  your  strategy  to  get  all  of  the  needed  information  while  maintaining 
your  secret  identity.  Each  form  has  a  Pre-Visit  Call  section  that  should  be  completed  before  visiting  the 
business.  This  is  a  good  opportunity  to  get  directions  to  the  facility,  verify  the  hours  of  operation  and  ask 
any  other  questions  that  will  help  you  complete  the  shop.  Make  sure  you  count  the  number  of  rings,  get  the 
employee’s  name  (even  if  you  have  to  ask)  and  so  on  down  the  form. 

Do  not  take  the  mystery  shopping  forms  with  you  into  the  business.  If  you  are  discovered,  then  the 
shop  is  no  longer  valid.  Do  not  stare  at  the  staff  and  then  write  down  notes  -  that  looks  very  suspicious! 
(Hint:  Take  a  3x5  note  card  with  you  and  take  specific  notes  while  you  perform  the  restroom  check.) 
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Ratings  And  Employee  Recognition  (Very  Important) 

For  rated  questions,  use  a  whole  number  between  1  and  5.  Consider  the  following  scale  when  rating 
these  questions: 

5  =  Outstanding  (The  way  it  should  be.  Please  use  them!  Don’t  save  5’s  for  perfection.) 

4  =  Good 

3  =  Satisfactory  (Nothing  special,  plain,  ordinary) 

2  =  Fair 

1  =  Poor  (Offensive,  negligent,  hazardous) 

One  of  the  main  goals  of  this  program  is  to  reward  our  employees  for  doing  it  right  -  providing 
outstanding  products  and  service  to  you,  our  customer.  Several  questions  ask  you  to  rate  the  performance 
of  employees.  An  employee  earning  a  “5”  will  receive  a  “caught  you  doin’  it  Wright!”  lapel  pin  to  proudly 
wear,  a  certificate,  and  one  paid  hour  off.  Please  do  not  penalize  a  great  performing  employee  for  items  out 
of  their  control.  A  delay  in  food  arriving  or  tasting  poorly  may  not  be  the  fault  of  the  server.  If  they  handle 
the  situation  with  professionalism  and  take  care  of  any  problems  please  reward  them  for  their  efforts! 

Always  get  a  name!  In  order  to  reward  the  right  employee,  it  is  essential  that  we  know  who 
provided  you  service.  All  employees  should  wear  nametags,  but  in  the  event  they  are  not,  casually  ask  their 
name.  (Hint:  If  you  use  their  name  later  in  the  conversation,  they  will  be  less  likely  to  suspect  that  you 
needed  their  name  for  another  purpose  such  as  mystery  shopping.) 

Always  Be  Professional 

As  mystery  shoppers  you  represent  the  Services  Division.  It  is  important  to  dress  appropriately  for 
the  facility  being  shopped.  The  idea  is  to  blend  in  as  a  “normal”  customer.  Please  avoid  wearing  shorts  at 
our  facilities  unless  it  is  appropriate  such  as  the  fitness  centers.  Please  use  good  judgement  and  follow  the 
rules  of  each  facility  for  appropriate  attire. 

It  is  important  that  you  do  not  draw  attention  to  yourself  as  a  shopper.  In  the  unlikely  event  that  you 
receive  inadequate  service  (i.e.  cold  food,  slow  service,  rade  service  etc.)  please  handle  it  in  a  low-keyed 
manner.  It  is  all  right  to  bring  the  problem  to  the  attention  of  your  server  so  they  can  correct  the  problem 
on  the  spot.  However,  please  do  not  demand  to  see  a  manager  and  seek  satisfaction  (unless  there  is  an 
immediate  safety  or  security  problem).  The  mystery  shopper  form  will  serve  this  purpose. 

You  must  tip  a  minimum  of  15%  at  all  table  service  restaurants.  If  the  service  was  exceptional, 
please  tip  more  accordingly,  but  a  maximum  of  20%  is  requested.  Even  if  you  were  very  impressed  or 
disappointed  with  the  service,  any  tip  above  20%  or  below  15%  would  draw  attention  to  yourself  as  a 
customer.  You  can  express  your  feelings  about  the  service  on  the  evaluation  form.  Important:  The  tip 
cannot  be  reimbursed  by  regulation.  As  a  result,  we  have  made  the  table  service  reimbursement  limit 
generous  to  help  compensate. 

The  shop  is  designed  to  measure  a  random  “normal”  customer  experience.  Please  do  not  test  the 
staff  by  asking  them  to  provide  unreasonable  service,  or  put  them  in  a  confrontational  situation.  Following 
the  questions  on  the  shopping  forms  and  instractions  is  a  good  way  to  avoid  this. 

Be  Flexible 

The  shopping  forms  are  designed  to  measure  the  specific  operations  performance  as  well  as  the 
overall  Services  Division’s  performance  in  certain  areas  such  as  cleanliness,  courtesy,  and  so  on.  The 
mystery  shopping  program  is  new,  so  it  is  possible  that  the  forms  may  not  match  up  exactly  with  each 
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activity  shopped.  For  example,  there  is  a  standard  fitness  center  shopping  form  for  the  3  fitness  centers  and 
two  civilian  health  clubs.  As  a  result,  a  question  about  the  aerobic  classes  may  not  directly  apply  to  every 
facility.  Instead  you  could  ask  the  employee  about  the  classes  offered  at  the  other  facilities,  obtain  a 
schedule,  directions  etc.  Please  remain  anonymous  and  continue  shopping  if  similar  situations  occur.  If  for 
some  reason,  you  are  not  able  to  complete  the  shop,  please  call  the  Marketing  Dept,  for  guidance. 

Be  Creative  and  Have  Fun! 

Many  of  the  shops  will  require  you  to  be  creative  to  obtain  the  necessary  information.  You  might 
shop  a  child  development  center  even  if  you  do  not  have  children,  or  make  reservations  for  lodging  as  if 
you  were  on  orders.  In  order  to  be  effective,  put  yourself  in  the  shoes  of  a  regular  customer.  Think  about 
the  questions  that  could  be  asked.  Obtaining  an  employee’s  name  (if  they  don’t  have  a  nametag)  can  seem 
awkward  and  difficult.  One  method  you  could  use  is  to  ask  if  the  employee  who  is  serving  you  is  named 
Bob  or  Sue  because  a  friend  told  you  that  he/she  provided  excellent  service.  They  will  normally  respond 
by  telling  you  their  name.  Then  you  can  quickly  say  that  you  must  have  remembered  the  wrong  name. 

You  could  say  they  looked  familiar  and  ask  what  their  name  is.  The  more  creative  you  are  the  more  fun 
and  easier  the  shopping  will  be! 

Shop  With  a  Friend 

It  is  more  enjoyable  (and  encouraged)  to  have  company  during  the  shopping  experience  -  it  also 
makes  you  less  conspicuous.  Take  along  your  family  or  a  friend  during  the  shop.  They  can  help  you 
remember  names,  times,  and  other  key  observations  while  completing  the  form.  Although  the  program  is 
set-up  for  one  mystery  shopper  per  shop,  many  of  the  shops’  reimbursable  limits  can  help  cover  part  of  a 
second  person’s  expenses.  For  example,  if  a  shopper  and  his/her  spouse  shop  the  bowling  center  and  the 
total  bill  is  $18.73,  a  total  of  $15.00  will  be  reimbursed.  If  the  same  shopper  shops  the  bowling  center 
alone  and  the  bill  is  $9.40,  only  $9.40  will  be  reimbursed.  Make  sure  that  whoever  you  shop  with  is  also 
eligible  to  use  the  facility,  and  that  all  expenses  are  included  on  the  same  receipt. 

Remain  Anonymous 

This  is  crucial  to  the  success  of  the  shop!  Please  do  not  complete  a  customer  comment  card  for 
your  visit.  It  could  be  matched  with  the  shopper  evaluation  and  identify  you  as  the  shopper.  All  of  your 
comments  can  be  included  on  the  shopper  form.  If  the  shop  is  not  going  well,  do  not  indicate  that  you  are  a 
mystery  shopper,  even  if  an  employee  directly  asks  if  you  are  a  mystery  shopper.  If  this  happens,  respond 
by  asking  what  a  mystery  shopper  is,  how  do  you  become  a  mystery  shopper,  and  if  you  get  your  med  free 
etc.  This  is  another  opportunity  to  be  creative. 

Completing  the  Form 

The  mystery  shopping  form  is  the  heart  of  the  program  (see  Attachment  2  for  completed  samples). 

It  is  essential  that  we  receive  accurate  and  thorough  information  about  our  programs  and  service.  Please 
read  and  answer  each  question  carefully.  Some  of  the  questions  have  a  Yes/No  answer  block  AND  a  rating 
section.  Make  sure  that  only  one  Yes/No  box  is  marked  per  question  and  that  all  box  marks,  ratings,  and 
comments  are  clear,  concise,  and  legible.  It  is  important  that  you  complete  the  form  immediately  after  the 
shop  while  the  information  is  fresh.  If  you  are  unable  to  do  so,  jot  down  important  notes  right  away 
(names,  times,  observations,  etc.)  as  a  minimum.  (Hint:  take  the  form  and  instractions  with  you  in  your  car 
so  you  can  look  at  them  before  and  after  the  shop.  Since  it  is  difficult  to  write  legibly  in  a  car,  write  down 
all  pertinent  notes  on  a  separate  piece  of  paper  and  transcribe  them  later  that  evening. 

The  comment  sections  are  very  important  in  explaining  the  good  areas  and  the  areas  needing 
attention.  Please  be  concise  in  the  comments  (no  ranting).  For  example,  if  you  had  to  wait  a  long  time  to 
be  served  write  “waited  14  minutes  to  be  served”  instead  of  “waited  forever  to  be  served”  and  so  on. 
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Conraients  are  required  explaining  all  responses  marked  as  “No”  or  rated  as  “2”  or  lower.  Please  use  the 
back  of  the  form  if  more  space  is  needed.  Sample  completed  forms  can  be  found  in  attachment  1. 

In  addition  to  explaining  your  ratings,  use  the  comment  sections  to  offer  suggestions  that  you  feel 
would  improve  the  business.  Maybe  there  is  a  menu  item  that  you  would  like  to  see,  or  a  special  program 
added.  For  example,  you  might  feel  strongly  about  replacing  the  Durkee™  brand  hot  sauce  with  Tabasco™. 
Please  do  not  feel  limited  by  the  questions  on  the  evaluation  form.  If  there  is  another  area  important  to  you, 
let  us  know  what  you  think! 

Lodging  is  Unique 

Shopping  the  lodging  operation  requires  an  overnight  stay.  As  a  result,  you  need  to  take  your 
evaluation  form  with  you  to  your  room  (but  not  the  front  desk!).  The  type  of  room  you  stay  in  (VOQ, 

VAQ,  DV  Suite,  etc.)  will  depend  on  the  next  room  type  that  needs  to  be  evaluated  and  the  availability  of 
rooms  on  the  night  you  are  staying.  Due  to  the  nature  of  obtaining  an  Air  Force  lodging  room,  you  will 
need  to  contact  the  lodging  manager  to  ensure  that  you  have  a  room  reserved  on  base.  Mystery  shoppers 
are  not  allowed  to  stay  in  contract  (off-base)  accommodations.  The  shopping  instructions  will  walk  you 
through  the  process  step  by  step.  If  you  wish  to  have  your  spouse  stay  with  you,  you  will  need  to  notify  the 
reservationist  when  you  call,  and  pay  the  extra  person  charge  (maximum  charge  is  $7).  This  charge  is  not 
reimbursable. 

Alcohol 

There  are  certain  restrictions  concerning  alcohol.  We  are  not  authorized  to  reimburse  any 
alcohol  purchase.  As  a  result,  you  must  make  sure  that  there  is  no  alcohol  included  on  your  itemized 
receipt  that  you  turn  in.  We  encourage  shoppers  over  21  years  of  age  to  sample  our  alcoholic  beverages  to 
let  us  know  how  we  are  doing.  (Hint:  If  you  would  like  an  alcoholic  drink  for  dinner,  please  order  and  pay 
for  it  in  the  lounge  before  hand  and  take  it  with  you  to  dinner.  This  will  keep  the  receipts  separate).  If  you 
are  at  the  bowling  center  or  golf  course,  purchase  your  alcoholic  drinks  separately  from  your  food. 

Obtaining  Reimbursement 

Make  sure  that  you  obtain  an  itemized  receipt  for  reimbursement  (a  credit  card  slip  doesn’t  qualify). 
This  allows  us  to  see  what  was  ordered  and  when.  The  receipt  is  required  in  order  to  obtain  reimbursement. 

Please  complete  and  return  the  mystery  shopping  form  to  the  Services  Marketing  Dept,  within  48 
hours  of  completing  the  shop  or  by  the  suspense  written  on  the  instructions  page  -  whichever  is  sooner. 
George  Parker,  Marketing  Director,  or  Sonya  Greene  (Bldg.  70,  Rm.  218  or  222),  will  review  your  form 
and  receipt,  and  authorize  the  correct  amount  for  reimbursement.  It  is  important  that  you  call  in  advance 
(257-8220)  to  schedule  a  time  to  return  the  completed  form.  This  will  ensure  that  you  receive  prompt 
service  and  help  the  marketing  staff  manage  their  workload.  It  will  take  at  least  15  minutes  for  the  staff  to 
review  your  form,  complete  the  reimbursement  process,  and  schedule  another  shop  (if  applicable). 

If  the  evaluation  form  is  turned  in  on  time,  legible,  and  completed  properly,  you  will  be 
presented  with  $10  in  Services  gift  certificates.  He/she  will  then  direct  you  to  the  Cashier’s  Cage  (just 
down  the  hallway)  for  reimbursement  of  your  authorized  expenditures.  Do  not  use  the  gift  certificates 
while  conducting  your  shops.  Services’  gift  certificates  can  not  be  used  to  purchase  alcohol  or  be  used  in 
conjunction  with  any  other  discount  or  promotion. 

While  you  are  in  the  building,  you  can  stop  by  Rm.  222  and  ask  Sonya  about  upcoming  shopping 
opportunities.  The  best  time  to  return  your  forms  is  between  8:00  a.m.  and  1:00  p.m.  Monday  -  Friday 
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when  the  Cashier’s  Cage  is  open.  The  Marketing  Dept  is  open  Monday-Friday  from  7:30  a.m.  -  4:40  p.m. 
Again,  please  call  in  advance. 

Help  Us  Recruit 

We  are  continuously  looking  for  new  shoppers  to  provide  us  with  a  fresh  set  of  eyes  to  look  at  our 
programs  and  services.  As  a  trained  shopper,  you  are  an  excellent  person  to  identify  other  people  who 
would  make  good  shoppers.  If  you  enjoy  mystery  shopping,  please  tell  others  about  our  program.  Give 
them  our  phone  number  or  recruiting  brochure  (available  in  Marketing)  so  they  can  contact  us  directly. 
Thanks  for  your  help! 

Independent  Contractor  Status 

As  a  Services  mystery  shopper,  you  are  an  independent  contractor  working  for  yourself.  Services 
provides  you  compensation  for  your  work  as  a  shopper  on  a  “one  shop  at  a  time”  basis.  This  provides 
flexibility  for  you  and  for  us.  You  can  stop  being  a  shopper  at  any  time.  If  you  decide  to  stop  shopping  for 
Services  please  give  the  Services  Marketing  Department  a  call  so  we  can  take  you  off  of  our  shopper  list. 
As  an  independent  contractor,  you  are  responsible  to  pay  any  payroll  taxes  that  may  apply.  Please  contact  a 
tax  advisor  if  you  need  more  information. 

Commercial  Opportunities 

If  you  find  that  you  really  enjoy  mystery  shopping,  there  are  numerous  commercial  companies  that 
hire  shoppers  for  local  businesses.  As  a  Services  shopper  your  training  and  experience  will  make  you  more 
marketable  with  these  companies.  If  you  are  interested  in  contacting  commercial  companies  a  good  place 
to  look  is  on  the  Internet.  Most  search  engines  will  find  hundreds  of  shopping  companies  by  searching  for 
“mystery  shopping.”  Another  resource  is  the  mystery  shopping  information  web  page  on  America  On- 
Line™  (no  federal  endorsement  intended)  at  http://inembers.aol.coin/msshoDnews/index.htm.  This  site  is  free  and 
has  links  to  numerous  mystery  shopping  companies.  Happy  shopping! 


If  you  decide  to  take  a  3  x  5  note  card  to  write  notes  while  visiting  the  restroom  here  are  some  reminders 
you  may  want  to  jot  down  on  one  side  of  the  card: 

REMINDERS: 

-  Call  for  directions 
Get  names 

-  Get  receipt  (no  alcohol  on  receipt) 

-  Look  and  act  professional 

-  Be  flexible  (and  creative) 

-  Ratings:  1  =  poor,  2  =  fair,  3  =  satisfactory,  4  =  good,  5  =  outstanding 

-  Complete  all  paperwork  completely  and  legibly 

-  Return  paperwork  to  Bldg.  70  by  the  suspense  (48  hours  max) 

-  Marketing  Office  phone:  257-8220 

. . .  And  anything  else  you  think  would  be  useful.  Be  sure  to  keep  this  card  out  of  view  of  any  employee! 


Attachment  1 


Mystery  Shopping  Instructions  -  Officers’  Club  Table  Dining 


General  Instructions: 

Location:  Building  800  Area  A 

Shop  must  be  completed  by:  Aug  26, 1998  .  Shop  any  day  between  Monday  and  Friday  .  Shop 
any  time  between  1830  and  2100  .  Return  your  completed  form  to  Services  Marketing  within  48  hours 
after  the  shop  or  by  the  completion  date  -  which  ever  is  sooner.  You  will  be  reimbursed  for  your  meal  expenses 
(maximum  reimbursement  is  $20.00).  If  the  form  is  completed  properly,  on  time,  and  legibly,  you  will 
receive  $10  in  Services  gift  certificates  to  use  in  any  of  our  activities. 


Read  the  questionnaire  thoroughly  before  shopping.  Think  about  wBjat  yqu^ll  need  to  do  to  answer  each 
question. 

Please  write  legibly  and  complete  the  form  with  black  or  blu^4nk  peiii  i 
Check  “Yes”  and  “No”  boxes  clearly.  Only  check  one  fort^ch  V  est 
For  rated  questions,  use  a  whole  number  betwe^ii,J^^n^|  |.  e 

r\  ^ 


questions: 


ibn.l 


5  =  Outstanding  (Tjre^ 
4  =  Good 


3  =  Satisfactory  (Nothing  sp^ci; 

2  =  Fair 

1  =  Poor  (Offensivfe,  negligent,  hazardous) 


eated 


jthiejsilrowing  scale  when  rating 
thbm!  Don’t  save  5’s  for  perfection.) 


iftr^rdinary) 


-  You  must  write  comments  explaining  all  “No”  answers  and  all  ratings  2  or  lower 

-  Make  sure  the  ratings  agree  with  the  written  comments. 

-  Keep  your  itemized  receipt  and  turn  it  in  with  the  completed  form!  Ask  for  receipt  if  one  is  not 
provided.  Alcoholic  beverages  will  NOT  be  reimbursed  and  must  NOT  be  included  on  the  receipt. 

-  Make  sure  you  obtain  the  employees’  names  including  the  employee  answering  the  phone. 

-  Coupons  received  for  mystery  shopping  cannot  be  used  to  pay  for  a  mystery  shop. 

-  Please  make  all  conmients  constructive  -  avoid  writing  your  opinions  too  harshly. 

-  Refer  to  your  “Services’  Guide  to  Mystery  Shopping”  or  call  Marketing  if  you  have  a  question. 


Specific  Instructions: 

-  Call  before  you  visit  the  restaurant.  The  number  is  257-9762.  Ask  the  employee  what  the  special  is,  when 
they  are  open,  and  get  directions. 

-  Visit  the  public  restrooms.  Evaluate  for  cleanliness  and  stocking  of  soap  and  paper. 

-  Evaluate  the  interior  and  exterior  of  the  Club. 

-  Purchase  a  full  meal  including  beverage.  Purchase  an  appetizer  or  dessert. 

-  Look  at  any  salad  bar,  taco  line,  deli  sandwich  line,  potato  bar,  etc.  even  if  you  don’t  order  fi-om  them. 

-  Ask  your  server  about  the  menu  and  specials  to  rate  employee  knowledge. 

-  Evaluate  the  general  service  at  other  tables  and  how  well  the  tables  are  bussed. 

-  It  is  essential  that  you  obtain  the  names  of  your  server  and  bartender,  ask  if  necessary. 

-  Don’t  forget  to  tip!  15%  is  a  good  rule  (Reference  your  Guide  to  Mystery  Shopping). 

-  Visit  Wings  (the  O’Club  bar)  and  purchase  a  drink.  Alcohol  will  not  be  reimbursed,  you  can  buy  a  non¬ 
alcoholic  drink,  but  you  are  encouraged  to  try  an  alcoholic  drink  and  let  us  know  how  we  are  doing. 


Tbanks  and  have  fun  shopping! 

Officers’  Club  Table  Dining  Instructions 


11/30/98 


Location: 


NO 


0 
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EVALUATION  mm  OEROe^S^OiUETAEt£  SEEVIOE 


Shopper  Name  or  Shopper#  | Arrival  Time  (24hr):  19:30  iDay  of  Week:  Frida 


:  20:50  IDate: _ 21  -Aua-98 


Ali  ratings  are  based  on  a  5-point  scale:  1=Poor,  2=Fair,  3=Satisfactory,  4=Good,  5=Outstanding. 
Write  comments  explaining  all  ratings  2  or  lower  and  all  NO  answers.  Write  on  back  if  necessary. 


Was  the  phone  answered  in  three  rings?  _ M  YES  Bno 


Did  the  person  answering  the  telephone  Identify  himself/herself  voluntarily?  0  YES  □  NO 


Name  of  person  answering  the  phone:  XENA 


Did  this  person  identify  the  business  name?  0  YES 


Did  this  person  have  a  friendly  demeanor  and  extend  a  greeting?  Please  rate. _ 


If  you  were  placed  on  hold,  how  long  did  you  have  to  wait  (in  seconds)?  38  Seconds 


Did  this  person  offer  directions  or  additional  information?  El  YES  QnO 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 


Comments:  The  phone  was  answered  in  four  rings.  Xeno  was  extremely  customer  friendly.  She  apologized 

for  putting  me  on  hold  ond  wos  very  helpful  in  explaining  directions  to  the  club^She  even  offered  to  tell  me  about 
their  evening  special.  r 


. 


Rate  the  exterior  appearance  of  the  facility  (cleanliness,  maint 


Rate  the  interior  appearance  of  the  facility  (cleanliness, 


Rate  the  atmosphere  (decor,  lighti 


Rate  the  condition  and  cleanline^  r 


Was  your  table  clean  and  attrac 


Was  the  floor  and  car 


Comments:  There  we 

nice  and  comfortable!  The  re 
pepper  shakers. 


□yes 

Bno 

Eyes 

□  no 

c  mitts  and  litter  around  the  front  entrance.  The  inside  decor  was  very 
didn’t  have  any  toilet  paper  in  one  stall.  There  were  food  stains  on  the  salt  and 


Server’s  name:  BRUNO 


and  braces 


Rate  the  overall  performance  of  the  server  (consider  courtesy,  accuracy  and  helpfulness). 

5 

Were  you  quickly  greeted  upon  arrival  and  made  to  feel  welcome? 

Byes 

□no 

Was  the  server  wearing  a  uniform  with  a  name  tag?  Was  it  neat  and  clean? 

Byes 

□no 

^■1 

Did  the  server  provide  friendly  and  helpful  service?  Please  rate. 

Byes 

□no 

5 

Were  the  menus  clean  and  in  good  condition? 

□yes 

Bno 

Was  your  server  knowledgeable  of  the  menu  and  specials  (if  any)?  Please  rate. 

Byes 

□  no 

5 

Did  the  server  provide  any  menu  selection  suggestions  (appetizer,  dessert,  etc.)? 

Byes 

□  no 

How  long  did  it  take  for  your  drinks  to  arrive  after  ordering? 

4  Minutes 

How  long  did  it  take  for  your  entrees  to  arrive  after  ordering? 

13 

Minutes 

hmmm 

Was  your  food  the  appropriate  temperature  upon  arrival? 

Byes 

□  no 

Was  your  table  and  others  around  you  properly  bussed? 

Byes 

□no 

Were  after  meal  drinks  suggested  by  the  server? 

Byes 

□no 

Was  your  guest  check  accurate? 

Byes 

□no 

v*vftv:v:vwxV: 
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Mm 

ssss 

u 

mm 

Did  the  anyone  from  the  staff  check  your  club  membership? 


□  yes 


Did  you  observe  a  manager?  Did  he/she  appear  to  be  involved  in  running  the  restaurant? 

□  yes  0NO 

Did  anyone  from  the  staff  thank  you  for  coming? 

Comments:  Bruno  was  excellent!  He  smiled  and  clearly  explained  the  specials.  He  recommended  the  salmon 

which  was  very  good.  The  salad  plate  was  very  warm.  There  were  food  stains  on  the  menu.  The  service  timing  was 
perfect,  the  meals  came  at  the  right  time.  We  were  not  asked  about  membership.  I  didn't  see  a  manager. 


Name  of  your  bartender  /  server: 

HERCULES 

Description  (sex,  height,  hair  color,  other): 

Male,  6*6",  brown  hair,  mu^ula 

Rate  the  overall  performance  of  the  server  (consider  courtesy,  accura 


Were  you  quickly  greeted  by  the  bartender/server?  _ /  ^ 


Was  the  bartender  /  server  wearing  a  uniform  with  a 


Did  the  bartender  /  server  smi! 


Did  the  bartender  /  server  asK  t(yS 


mely  manner? 


0YES 


□yes 


0YES 


0YES 


0YES 


Did  you  receive  your  drink(s 


Was  your  bar  bill  accurate? 


Comments:  I  had  to  wait  5  minutes  to  be  greeted.  The  bartender  was  not  wearing  a  nametag.  The 

specialty  drink  was  well  made.  Hercules  didn't  suggest  any  drinks  or  food.  The  bar  was  very  inviting  and 
comfortable.  I  suggest  the  bar  sell  Bitburger  Beer  if  possible  (o  lot  of  people  have  been  stationed  in  Germany). 


Rate  the  overall  quality  of  service  provided  bv  the  staff 


Rate  the  perceived  value  for  the  dollar 


Rate  the  overall  cleanliness 


Rate  the  overall  Officer’s  Club  experience 


Based  on  THIS  experience,  would  you  return?  I3yes  Dno 


Overall  comments:  The  dirTy  menu  and  salt  and  pepper  shakers  took  awoy  from  a  generolly  clean  facility.  It  would 
be  nice  to  see  more  healthy  choice  items  on  the  menu.  Thanks! 
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Shopper: 

Name  or  Shopper  # 

Room  #: 

4212  Bldg  833 

Appendix  K 


msTEm  iVAtuATON  worn 


Arrive  Time:  16:45  [Arrive  Date:  06-Aug-98 


Attachment  2 


All  ratings  are  based  on  a  5-point  scale:  1=Poor,  2=Fair,  3=Satisfactory,  4=Good,  5=Outstanding. 
Write  comments  explaining  all  ratings  2  or  lower  and  all  NO  answers.  Write  on  back  if  necessa 


EIyes 


□  yes  0NO 


NO 


Was  the  phone  answered  in  three  rings? 


Did  the  employee  answering  the  phone  identify  himself/herself  voluntarily? 


Name  of  employee  answering  the  phone:  RACHAEL 


Did  this  person  identify  their  work  center  (i.e.  front  desk  or  reservations)? 


Did  this  person  have  a  friendly  demeanor  and  extend  a  greeting?  Please  rate. 


Were  directions  or  additional  information  offered? 


If  you  were  placed  on  hold,  how  long  did  you  have  to  wait  (in  seconds)? 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness’ 


Comments:  When  colling  reservations,  I  was  put  on  hold  for  4  1/2  minutes.  No  additional  info  was  offered 

by  Rachael.  I  had  to  ask  directions  to  the  lodging  and  when  I  needed  to^eck*in  by. 


0YES  UNO 


0YES  GnO 


□yes  JTIno 


270  Seconds 


Rate  the  exterior  appearance  of  the  facility  (cleanliness,  maintenanc 


Rate  the  interior  appearance  of  lobby,  hallways,  and  stairwell 


Rate  the  lobby  atmosphere  (decor,  lighting,  temper 


Rate  the  condition  and  cleanli 


Was  the  laundry  room  conv 


Were  the  ice  machines  con 


Were  the  parking  areas  clea 


Were  directional  signs  to 


Comments:  The  h 


ive  and  easv  to  follow? 


was  dark  with  no  woll  art.  The  hallway  carpets  were  very  stained  and  old. 

There  was  a  lot  of  dirt  and  discarded  laundry  boxes  behind  the  washer.  One  set  of  machines  is  not  adequate  for  40 
rooms.  The  automatic  door  to  the  lobby  was  not  working  making  it  difficult  to  enter  and  leave. 


en’s  □  Women's 


ble  and  hanging  rack? 


mmmm 


Name  of  front  desk  employee  who  checked  you  in:  KE&GIE 


Description  (sex,  height,  hair  color,  approx  age):  Male,  6*1*',  dark  brown  hair,  around  30  yrs  old,  glasses 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 


Did  the  clerk  ask  if  you  needed  any  information  about  the  base  or  local  area?  □  YES  [3n0 


Name  of  front  desk  employee  who  checked  you  out:  6RETA 


Description  (sex,  height,  haircoior,  approx  age):  Female,  5'3'’.  redish  brown  shoulder  length  hair,  26  yrs  old,  freckles 


Rate  the  overall  performance  of  this  employee  (consider  courtesy,  accuracy  and  helpfulness 


Was  your  bill  accurate? 


Were  you  quickly  greeted  upon  arrival  at  the  front  desk  each  time  and  made  to  feel  welcome? 


Were  staff  members  in  proper  uniforms  with  nametags?  Were  they  neat  and  clean? 


Did  the  clerk(s)  provide  friendly  and  helpful  service?  Please  rate. 


Were  staff  members  knowledgeable  about  their  jobs,  the  base  and  local  area?  Please  rate. 


Was  the  check-in/out  process  quick  and  efficient?  (1  min.  for  check-in,  3  min.  for  check-out 


Comments:  Reggie  didn't  ask  if  I  needed  any  infor*mation  about  the  base  or  locol  area.  He  and  another 

employee  weren't  wearing  nametags.  6reta  was  very  nice  and  courtious.  She  asked  how  my  stay  was  and  smiled  a 
lot.  There  were  several  customers  and  phone  calls  and  she  handled  it  very  well! 


[3yes  Dno 


[ZIyes  Dno 


IZIyes  nNo 


IZIyes  Dno 


IZIyes  Hno 
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Was  the  carpet  clean  and  free  of  stains  and  wear  marks? 


Was  the  furniture,  counters,  ledges,  etc,  free  of  dust?  _  _ 


Was  the  bathroom  clean?  Did  it  smell  fresh?  Was  the  toilet  clean  inside  and  out? 


Was  the  shower  /  bathtub  clean?  (free  of  soap  scum  and  mold?  clean  shower  curtain,  etc»?) _ 


Were  the  sink  and  vanity/mirrors  clean?  (Including  soap  dish  and  counter  if  applicable 


Were  the  microwave  and  refrigerator  clean?  (Check  the  inside  door  and  ceiling.) _ 


Were  the  towels  (two  sets)  and  linens  clean  and  in  good  condition? 


Was  the  bed  well  made? 


Were  amenities  provided?  (Shampoo  with  conditioner,  mouthwash,  lotion  and  shoe  mitt. 


Were  quality  brand  facial  and  toilet  tissue  provided?  (2  ply  is  standard 


Were  deodorant  soap  and  facial  soap  provided?  (Quality  commercial  product,  1.25  oz. 


Was  the  housekeeper  wearing  a  uniform  with  a  name  tag?  Was  it  neat  and  clean? 


Name  of  your  housekeeper:  SAMANTHA 


Rate  the  performance  of  your  housekeeper  (room  cleanliness,  stocking,  and  courtes 


Comments:  Soap  dish  was  rusty.  Shower  has  mold  in  the  corners.  Shower  curtain  has  a  tot  of  mold  stains. 

Microwave  was  dirty  on  the  inside  of  the  door  and  ceiling.  No  lotion  or  sho^^itt  wprpr^vided.  There  was  dust  on 
the  plants  and  pictures.  There  was  dirt  in  the  chair  cushion.  There  is  a  2*  tear/fe/ffte  r4om‘s  ceiling  cover. 


My  spouse  stayed  with  me,  but  there  was  only  one  set  of  towels  in  t 


. ^ . ^ 


Was  the  guest  information  book  in  good  condition?  _ 


Did  the  quest  information  book  provld^^seful  an 


Were  the  TV,  remote  control,  V 


Were  clear  instructions  provide 


El  YES 

□no 

El  YES 

□no 

El  YES 

□  no 

□yes 

El  NO 

El  YES 

□  no 

□yes 

El  NO 

□yes 

El  NO 

El  YES 

□no 

El  YES 

□no 

El  YES 

□no 

El  YES 

□no 

El  YES 

□no 

xondition? 


em  skirt  hangers? 


Was  there  an  iron  and  ironing 


Were  ten  hangers  provided?  W 


Did  the  room  have  adequate  liah 


Was  everything  in  your  room  in  working  order? 


Did  the  furnishings  and  decor  compliment  each  other? 


Were  the  furnishings  in  good  condition  and  well  placed? _ 


Were  the  drapes  in  good  condition?  Did  they  block  all  outside  light  when  closed? 


Were  there  working  light  bulbs  in  every  lamp  including  3-way  bulbs  in  3-way  lamps? 


ood?  Did  the  bathroom  exhaust  fan  work  properly? 


Were  the  spare  blankets  and  pillows  wrapped  in  plastic? 


Was  the  room  stocked  with  resale  items?  Was  it  a  good  selection? 


Comments  (including  resale  suggestions):  There  were  food  stains  In  the  information  book.  Community  Center  no 

longer  at  Kitthawk  Center.  The  furniture  and  decor  are  old,  worn,  and  dark.  Spare  blanket  not  wropped  in  plastic. 
A  gin  bottle  was  already  open  and  filled  with  water.  1  would  prefer  bottled  Juice  over  the  cans. 


PYES 

0NO  1 

El  YES 

□no 

El  YES 

□  no 

El  YES 

□  no 

El  YES 

□no 

El  YES 

□no 

El  YES 

□  no 

□yes 

EIno 

El  YES 

□no 

El  YES 

□no 

El  YES 

□no 

□yes 

EIno 

El  YES 

□no 

Rate  the  overall  room  qual 


Rate  the  overall  quality  of  service  provided  by  the  lodging  staff 


Rate  the  perceived  value  for  the  dollar 


Rate  the  overall  cleanliness 


Rate  your  overall  lodging  experience 


Based  on  THIS  experience,  would  you  return?  1 0  YES  [H  NO 


Overall  comments:  There  is  a  lorge  wet  area  in  front  of  the  refrigerator  on  the  floor.  The  bed  had  two  pillows 
one  was  too  large  and  the  other  was  too  small  --  very  uncomfortable,  and  the  bed  looked  lopsided  when  made. 
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Mystery  Shopper  Demoeraphic  Survey 


Please  complete  the  following  questions  as  accurately  and  completely  as  possible  so  we  can 
match  your  customer  profile  with  the  right  Services’  activities  to  assign  mystery  shops.  The 
information  you  provide  will  be  used  solely  for  the  Services’  Mystery  Shopping  program  and  will  be 
destroyed  when  you  leave  the  program.  All  information  provided  will  be  kept  confidential  and  used  in 
accordance  with  the  Privacy  Act  of  1974. 


Last  Name: _  First  Name: _  Grade/Rank: _ 

Address: _  City: _  St: _ Zip: _ 

Home  Phone:  (  ) _ Work  Phone:  (  ) _ Contact  you  at  work?  Y  /  N 

Complete  E-mail  address: _  Office  Symbol: _ 

Category  (check  all  that  apply): 

I  I  Officer  Q  Enlisted  Q  Contractor  Q  Reservist  I  lOther: _ 

I  I DOD  Civilian  Q  Retiree  Q  Family  Member  -  Sponsor’s  Category?: _ 

Age  Category:  □  Under  21  □22-35  □  36-49  □  50  and  Over 

Gender:  □  Male  □  Female 


Are  You  Married?  Y  /  N  #  of  Children  under  18  years  old?  (Please  list  their  gender  and  ages): 


Please  check  the  days  and  times  that  you  would  normally  be  available  to  perform  mystery  shopping: 
Monday  -  Friday:  □  Mornings  (Breakfast)  □  Mid-Day  (Lunch)  □  Evenings  (Dinner) 

Saturday  -  Sunday:  □  Mornings  (Breakfast)  □  Mid-Day  (Lunch)  □  Evenings  (Dinner) 

Are  you  available  to  stay  overnight  in  Lodging?  Y  /  N 


Please  check  the  Services  activities  you  have  used  during  the  past  year  or  are  interested  in  using. 

I~1  Skills  Development  Center  O  Enlisted  Club  O  Lodging 

I  I  Auto  Skills  Center  □  Flywright  Club 

[~~l  Prairie  Trace  Golf  Course  (Military)  CH  Officer’s  Club 

r~1  Twin  Base  Golf  Course  (Civilian)  □  Pitsenbarger  Dining  Facility 

□  New  Horizon  Child  Dev.  Center  (CDC)  □  Kittyhawk  Bowling  Center 

O  Kittyhawk  CDC  O  Outdoor  Recreation 

I  I  Page  Manor  CDC  Q  Library 

Q  Wright  Care  CDC  □  Veterinary  Clinic 

n  Tots  In  Blue  □  Information,  Ticket  and  Tours 

□  Preschool  (on  base)  □  Riding  Club 

□  Youth  Center  □  Tennis  Club 

Q  Community  Activity  Center  □  Human  Resource  Office 

n  Base  Restaurants  (please  list  the  locations  you  frequent  the  most): _ 


□  Civilian  Recreation 
r~l  Rod  and  Gun  Club 
r~l  Aero  Club 

Q  Dodge  Gym 

□  Hilltop  Gym 
r~l  Jarvis  Gym 

□  Wright  Field  Fitness  (Hangar  6) 

□  Health  Club  A 

□  Health  Club  B 
n  Aquatics 

f~l  Recycling  Center 


Please  list  the  activities  you  are  a  member  of  (i.e.  Officers’  Club,  Health  Club  A,  Aero  Club,  etc.): 


Do  you  have  any  previous  mystery  shopping  experience?  Y  /  N  (If  yes,  please  describe  briefly) 


APPENDIX  M:  Mystery  Shopper  Database 


Appendix  M  1 1/30/98 


(6 

v .  .c 

::  O 

««» 

■■■  ""Ol 

:  c 
o 

•  S: 

E 

o 

§ 

Vi:.  .-a 
:■■■■' ^;0 
,■■2: 

O'Club  and  O'club  fitness  center  member  | 

£ 

j 

s 

s 

1 

0 

Jl 
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O'Club  member,  No  bowling  or  golf  I 

rsj 

z 

11 

i 

E 

5 

1 

Lots  of  activities  I 

0 

X 

X 

Aero  Club,  Rod  and  Gun,  All  fitness  -  Yes  | 
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ftkjVIystery  Shopping  Program  Critique 


The  88*  Services  Division  Mystery  Shopping  program  has  recently  been  developed.  I  am 
working  on  this  project  for  part  of  my  Master  Degree  requirements  at  Purdue  University.  Complete  this 
form  and  return  it  to  the  Services  Marketing  Department  with  your  completed  mystery  shopping 
evaluation  form.  Your  detailed  answers  and  suggestions  will  be  used  to  improve  the  program  for  the 
future  and  help  a  struggling  student  graduate  ©.  Thank  you. 


Justin  W.  Hall,  Capt,  US  AF 
Graduate  Student,  Purdue  University 


1 .  Was  the  training  information  effective  in  preparing  you  to  mystery  shop?  Yes  /  No 

2.  Did  the  training  techniques  help  you  learn  the  information?  Yes  /  No 

3.  What  would  you  suggest  be  changed  about  the  class? 


4.  Were  the  shopping  instractions  useful  and  easy  to  follow?  Yes  /  No 

5.  What  would  make  the  instructions  easier  to  follow  and  more  useful? 


6.  Were  the  evaluation  forms  effective  for  the  shop?  Yes  /  No 

7.  What  would  make  the  evaluation  forms  better? 


8.  Was  the  Services  Guide  to  Mystery  Shopping  a  useful  reference?  Yes /No 

9.  What  would  you  suggest  to  improve  the  Guide? 


10.  Has  this  program  positively  affected  your  opinion  of  the  88*  Services  Division?  Yes  /  No 


Thanks  for  your  feedback! 
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Mystery  Shopping  Program  Critique  Consolidated  Results 

Total  Shops:  43  Critiques  completed:  36  Critiques  with  comments:  23 

Was  the  training  information  effective  in  preparing  you  to  mystery  shop?  35  "Yes” 
responses,  1  non-response 

-  I  thought  it  covered  all  of  my  questions  (2)* 

-  Yes,  but  I  don’t  think  anything  can  truly  prepare  you  except  experience 

-  Excellent  job,  very  informative  (4) 

-  Help  us  understand  the  88*  Services  better.  Use  the  Services  web  page  to  explain  where  the 
shopping  places  will  be.  Give  everyone  the  URL.  This  is  a  great  opportunity  to  explain 
more  about  services. 

Did  the  training  techniques  help  you  learn  the  information?  35  "Yes”  responses,  1  non¬ 
response 

-  Role  play  in  training  class  how  to  get  an  employee’s  name 

-  It  was  enjoyable 

-  I  thought  the  class  was  good  and  very  helpful  (5) 

-  Maybe  bring  in  some  experienced  shoppers  to  the  next  class  to  share  their  experiences 

-  Better  mixed  group  - 1  was  the  only  guy  and  the  only  reservist 

Were  the  shopping  instructions  useful  and  easy  to  follow?  36  "Yes”  responses 

-  Making  the  lodging  reservation  was  difficult,  need  better  instructions  (2) 

-  The  instructions  were  clear  and  straightforward  (4) 

-  Can’t  think  of  anything  -  you  did  a  great  job  (2) 

-  Make  them  more  specific  about  the  shop 

-  For  the  most  part  all  of  the  questions  did  not  apply  -  museum  canteen 

-  48  hours  notice  is  insufficient  time  for  a  Friday  night  shop 

-  Number  on  the  lodging  form  was  DSN  instead  of  commercial 

-  Lodging  manager  wasn’t  aware  how  the  bill  was  going  to  be  handled  -  created  confusion 

Were  the  evaluation  forms  effective  for  the  shop?  36  "Yes”  responses 

-  Separate  cleanliness  from  other  standards.  It’s  a  double  question,  which  could  be  half- 
correct  and  half-wrong  at  the  same  time. 

-  Evaluation  forms  should  be  modified  to  better  fit  snack  bars  instead  of  just  restaurants 

-  Too  many  yes/no  questions.  For  example  did  the  employee  have  a  friendly  demeanor  and 
extend  a  greeting.  This  is  a  double  question.  If  the  employee  only  did  one  of  these  then 
“No”  would  have  to  be  checked  and  a  comment  written. 

-  They  are  great  the  way  they  are  (4) 

-  Seemed  a  little  more  suitable  for  O’Club  type  facility 

-  Number  each  section  -  easier  to  refer  to  comments 

-  More  specific  -  i.e.  no  waiters  or  servers  at  this  facility  -  Flywright  Club  (2) 

-  For  the  most  part  all  of  the  questions  did  not  apply  -  museum  canteen 

-  Except  no  restrooms  to  check  -  canteen  #2 

-  At  least  one  question  on  handicap  access/facilities  would  be  nice  (and/or  other  impairments 
such  as  for  old  age). 


*  Number  of  repeat  responses  in  parentheses 
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-  O’Club  lunch  is  a  buffet,  form  is  slanted  toward  dinner 

-  It  would  be  helpful  to  get  the  forms  on  a  disk  so  the  responses  could  be  t5^ed  (2) 

-  There  was  no  server  to  greet  me  until  I  ordered  from  the  food  line 

-  The  question  “Were  you  quickly  greeted  upon  arrival  and  made  to  feel  welcome?”  under 
“Service”  could  be  left  off  or  reworded.  Maybe  you  could  use,  “Did  the  server  take  your 
order  promptly?” 

-  Since  the  server  on  the  line  gave  me  the  food  as  soon  as  I  requested  it,  there  was  no  wait  to 
be  served.  Therefore,  “How  long  did  it  take  before  you  were  served”  could  also  be  taken  off 
or  reworded.  Maybe  you  could  use,  “Did  you  have  to  wait  long  in  the  food  line  before  being 
served?” 

-  Add  the  time  for  the  phone  call 

-  Add  the  time  for  in/out  bar  and  dining 

-  Add  an  evaluation  of  the  web  page  information.  This  is  a  key  source  of  information  like  the 
phone  call. 

-  Some  questions  may  be  more  informative  with  a  rating  too. 

Was  the  Services  Guide  to  Mystery  Shopping  a  useful  reference?  36  **Yes^*  responses 

-  The  guide  is  well  organized  and  very  useful!  (6) 

-  Issue  a  small  binder  or  folder  to  keep  shop  guides  and  other  items  together  and  organized 

-  Add  the  Services  web  URL  and  make  the  link  easier  to  find  on  the  public  ASC  and  APMC 
pages 

Has  this  program  positively  affected  your  opinion  of  the  88‘‘‘  Services  Division?  30  “Yes’* 
responses,  6  non-responses 

-  I  never  realized  Services  was  involved  in  so  many  programs 

The  fact  that  they  care  enough  to  bother  with  the  program  shows  they  care.  This  program  is 
a  big  undertaking. 

-  Now  I  think  they  care,  but  I  will  wait  for  results 

-  I’m  glad  they’re  trying  to  have  more  friendly,  helpful  employees 

-  This  is  a  great  program  for  two  reasons:  1 .  I  will  be  going  to  places  on  base  that  I’ve  never 
been.  2.  The  feedback  can  be  used  to  reward  employees  and  improve  service 

-  It  shows  you  are  trying  and  that  you  not  only  want  to  provide  good  service,  but  that  you  want 
to  reward  your  folks. 

Other  Comments: 

-  Spread  the  word  on  base 

-  Services  Web  page  needs  to  be  improved 

-  Suggest  you  put  a  few  coupons  on  the  web  page  (like  you  do  with  the  cafeteria  specials 
coupon  for  25  cents  off) 


*  Number  of  repeat  responses  in  parentheses 
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Bowling  Center _ Jessie  Atwell _ 5  14-Aug-98  17:00  Short,  medium  weight,  short  blonde  hair 

Bowling  Center _ Linda  Rolph _ 5  14-Aug-98  17:00  Employee  Answering  Telephone _ 

Bowling  Center _ Steve  Reynolds _ 5  14- Aug-98  17:00  Male _ 

Bowling  Center _ Norma _ 4  8-Sep-98  1 1 :40  Female,  5'3”,  Red  hair 
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Lodging _ Sgt  Richard  Robinson _ 5  10“0ot“98  16:54  Employee  Answering  Telephone 

Lodging _ 10>Oct-98  16:54  Housekeeper  for  room _ 

Lodging _ 3  10-Oot-98  16:54 _ 

Lodging _ 3  10-Oct-98  16:54  Female,  Asian,  Dark  hair,  40*s 

Officers*  Club _ Hope  Dupree _ 5  21-Aug-98  12:15  Female,  57”,  black _ 
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Restaurant  #1 _ Cashier _ 3  15-Sep-98  11:30 _ 

Restaurant  #1 _ Larry _ 4  15-Sep-98  11:30  Employee  Answering  Telephone _ 

Restaurant  #1 _ Wanda _ 3  15-S6p-98  11:30  Dark  hair _ 

Restaurant  #2 _ Nanny _ 4  24- Aug-98  11:10  Female,  brown  hair,  around  57*,  maybe  late  40's 

Restaurant  #2 _ Nanny _ 4  24-Aug>98  11:10  Female,  brown  hair,  around  57^  maybe  late  40's 


Employee  Answering  Telephone 

Employee  Answering  Telephone 

Male,  approx  5'9\  dark  brown/black,  wore  chef  hat  and  glasses  I 

Female,  approx  5*10"  and  160lbs,  dark  brown  med  length  hair,  pink  glasses 

Male,  5'10",  Brown  hair  -  balding  | 

Female,  5'8",  Graying  I 

Female,  she  was  sitting  but  looked  small,  brown  hair 

Female,  5‘3",  Brown,  mid  age 

Employee  Answering  Telephone 

Short  dark  hair 

Asian,  short  black  hair,  glasses,  female  I 

Employee  Answering  Telephone  I 

Female,  Dark  hair  | 

Employee  Answering  Telephone  I 

Female,  5'3“,  sandy  hair,  blue  eyes 

Male,  5'8",  brown  hair 

Employee  Answering  Telephone 
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Sounded  like  “Nanny"  | 

Nan  I 

Unknown  •  No  Nametag 

Unknown  --  No  Nametag 

Cook  I 

Naniy  Master 

NannieMaster  | 

Debra  | 

Geraldine  | 

Tol  1 

Debra  Rickard  | 

Toi  1 

Asoka  Munaweera  (Betty) 

Daniel  | 

Elena  Russel  | 

Heather  Albright  | 
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